We're running nginx 0.7.68 w/ the s3-proxy patch applied, and we're seeing reproducible segfaults when we hit (or miss) certain files:<div><br></div><div><div>Nicholas-Tangs-MacBook-Air:~ ntang$ curl http://[server]/path/to/test.txt</div>
<div>curl: (52) Empty reply from server</div></div><div><br></div><div><br></div><div>And here's what happens on the server:</div><div><br></div><font class="Apple-style-span" face="'courier new', monospace">[root@server nginx]# strace -p 9742<br>
Process 9742 attached - interrupt to quit<br>write(26, "2010/12/20 11:23:52 [info] 9742#"..., 84) = 84<br>epoll_wait(5, {{EPOLLIN, {u32=857063440, u64=139771977777168}}}, 512, 4294967295) = 1<br>accept(6, {sa_family=AF_INET, sin_port=htons(44473), sin_addr=inet_addr("myip")}, [29640703181062160]) = 3<br>
ioctl(3, FIONBIO, [1]) = 0<br>epoll_ctl(5, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLET, {u32=857064145, u64=139771977777873}}) = 0<br>epoll_wait(5, {{EPOLLIN, {u32=857064145, u64=139771977777873}}}, 512, 5000) = 1<br>
recvfrom(3, "GET /path/to/..."..., 1024, 0, NULL, NULL) = 210<br>stat("/path/to/file/test.txt", 0x7fff3b23df20) = -1 ENOENT (No such file or directory)<br>epoll_ctl(5, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=857064145, u64=139771977777873}}) = 0<br>
--- SIGSEGV (Segmentation fault) @ 0 (0) ---<br>Process 9742 detached<br>[root@server nginx]#</font><br><div><div> <span style="text-indent: 0px; "><div style="word-wrap: break-word; "><span style="text-indent: 0px; "><div style="word-wrap: break-word; ">
<span style="text-indent: 0px; "><div style="word-wrap: break-word; "><div style="font-family: Helvetica; font-size: medium; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; font-size: medium; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Has anyone seen any similar behavior? We've got a series of location/ rewrite/ try_files blocks to look for the files. Basically, the way it works is like this:</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
- look on the main filesystem</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
- look on the backup filesystem</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
- look on the backup backup filesystem</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
- look on S3</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
In the case of the request above, the process segfaulted and died after step 1.</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Normally, this is what should happen:</div><div style="font-family: Helvetica; font-size: medium; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN, {u32=857063440, u64=139771977777168}}}, 512, 4294967295) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">accept(6, {sa_family=AF_INET, sin_port=htons(46329), sin_addr=inet_addr("my_ip")}, [29640703181062160]) = 3</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">ioctl(3, FIONBIO, [1]) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLET, {u32=857063792, u64=139771977777520}}) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN, {u32=857063792, u64=139771977777520}}}, 512, 5000) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(3, "GET /path/to/request"..., 1024, 0, NULL, NULL) = 210</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">stat("/path/1/test.txt", 0x7fff3b23df20) = -1 ENOENT (No such file or directory)</font></div><div><font class="Apple-style-span" face="'courier new', monospace">stat("/path/2/test.txt", 0x7fff3b23dd90) = -1 ENOENT (No such file or directory)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">stat("/path/3/test.txt", 0x7fff3b23dc00) = -1 ENOENT (No such file or directory)</font></div><div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=857063792, u64=139771977777520}}) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 7</font></div><div><font class="Apple-style-span" face="'courier new', monospace">ioctl(7, FIONBIO, [1]) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=857063969, u64=139771977777697}}) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">connect(7, {sa_family=AF_INET, sin_port=htons(8976), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLOUT, {u32=857063792, u64=139771977777520}}, {EPOLLOUT, {u32=857063969, u64=139771977777697}}, {EPOLLIN, {u32=857063440, u64=139771977777168}}}, 512, 60000) = 3</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(3, 0x7fff3b23e1f7, 1, 2, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)</font></div><div><font class="Apple-style-span" face="'courier new', monospace">getsockopt(7, SOL_SOCKET, SO_ERROR, [3681060957225746432], [4]) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">writev(7, [{"GET /path/to/request"..., 203}], 1) = 203</font></div><div><font class="Apple-style-span" face="'courier new', monospace">accept(6, {sa_family=AF_INET, sin_port=htons(40347), sin_addr=inet_addr("127.0.0.1")}, [51539607568]) = 8</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">ioctl(8, FIONBIO, [1]) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLET, {u32=857064145, u64=139771977777873}}) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN, {u32=857064145, u64=139771977777873}}}, 512, 5000) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(8, "GET /path/to/request"..., 1024, 0, NULL, NULL) = 203</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_MOD, 8, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=857064145, u64=139771977777873}}) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 9</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">ioctl(9, FIONBIO, [1]) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">epoll_ctl(5, EPOLL_CTL_ADD, 9, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=857064321, u64=139771977778049}}) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">connect(9, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("s3_ip")}, 16) = -1 EINPROGRESS (Operation now in progress)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLOUT, {u32=857064145, u64=139771977777873}}}, 512, 60000) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(8, 0x7fff3b23e1f7, 1, 2, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLOUT, {u32=857064321, u64=139771977778049}}}, 512, 59999) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">getsockopt(9, SOL_SOCKET, SO_ERROR, [3681062469054234624], [4]) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">writev(9, [{"GET /path/to/request"..., 332}], 1) = 332</font></div><div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN|EPOLLOUT, {u32=857064321, u64=139771977778049}}}, 512, 59998) = 1</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(9, "HTTP/1.1 200 OK\r\nx-amz-id-2: [snip]"..., 4096, 0, NULL, NULL) = 447</font></div><div><font class="Apple-style-span" face="'courier new', monospace">readv(9, [{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 3649}], 1) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">close(9) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">writev(8, [{"HTTP/1.1 200 OK\r\nServer: nginx/0"..., 442}, {"bla\n", 4}], 2) = 446</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">write(27, "127.0.0.1 - - [20/Dec/2010:11:27"..., 190) = 190</font></div><div><font class="Apple-style-span" face="'courier new', monospace">close(8) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN|EPOLLOUT, {u32=857063969, u64=139771977777697}}}, 512, 59987) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(7, "HTTP/1.1 200 OK\r\nServer: nginx/0"..., 4096, 0, NULL, NULL) = 446</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">readv(7, [{"\0\0\300\220k\0\0\0\0\0\310\0\0\0\0\0\0\0\6\0\0\0\0\0\0\0\270\240k\0\0\0"..., 3650}], 1) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">close(7) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">writev(3, [{"HTTP/1.1 200 OK\r\nServer: nginx/0"..., 470}, {"bla\n", 4}], 2) = 474</font></div><div><font class="Apple-style-span" face="'courier new', monospace">write(27, "my_ip - - [20/Dec/2010:"..., 219) = 219</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(3, 0x69e5c0, 1024, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">epoll_wait(5, {{EPOLLIN|EPOLLOUT, {u32=857063792, u64=139771977777520}}}, 512, 5000) = 1</font></div><div><font class="Apple-style-span" face="'courier new', monospace">recvfrom(3, "", 1024, 0, NULL, NULL) = 0</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">write(26, "2010/12/20 11:27:56 [info] 10313"..., 92) = 92</font></div><div><font class="Apple-style-span" face="'courier new', monospace">close(3) = 0</font></div>
</div><div style="font-family: Helvetica; font-size: medium; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Here's how the config lines work:</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space: pre; ">        </span># Files served from Set A</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>location ~ ^/dv0([\d])/(.*)$ {</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>alias<span class="Apple-tab-span" style="white-space:pre">                </span>/path1/$1b/path/$2;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>try_files <span class="Apple-tab-span" style="white-space:pre">        </span>"" /dv$1b/$2;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>}</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br>
</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span># Files served from Set B</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>location ~ ^/dv0([\d])b/(.*)$ {</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>alias<span class="Apple-tab-span" style="white-space:pre">                </span>/path2/$1/path/$2;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>try_files <span class="Apple-tab-span" style="white-space:pre">        </span>"" /union/$2;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>}</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br>
</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span># Files served from the Union of A and B</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> location ~ ^/union/path3/(.*)$ {</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> <span class="Apple-tab-span" style="white-space:pre">        </span>alias<span class="Apple-tab-span" style="white-space:pre">                </span>/union/path3/$1;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> <span class="Apple-tab-span" style="white-space:pre">        </span>try_files<span class="Apple-tab-span" style="white-space:pre">        </span>"" @failUnion;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> }</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="'courier new', monospace">        </font></span></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>location @failUnion {</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> rewrite ^/union/path/(.*)$ /$1 break;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> proxy_pass <a href="http://127.0.0.1:8976">http://127.0.0.1:8976</a>;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> }</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>location ~ ^/path/(.*)$ {</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>rewrite ^/path/(.*)$ /union/path3/$1 last;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span>}</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">        </span># Proxy the request back to Amazon S3</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> location / {</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> <span class="Apple-tab-span" style="white-space:pre">        </span>proxy_pass<span class="Apple-tab-span" style="white-space:pre">        </span><a href="http://bucket.s3.amazonaws.com">http://bucket.s3.amazonaws.com</a>;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>proxy_s3_auth on;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>proxy_s3_secure_download off; # optional if you're also using secdownload</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>proxy_s3_bucket bucket;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>proxy_s3_user user;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre">                </span>proxy_s3_pass pass;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> }</font></div>
</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Hopefully that gives you some idea of what we're doing without revealing anything confidential... ;)</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br></div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Thanks,</div><div style="font-family: Helvetica; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
Nicholas</div><div style="font-family: Helvetica; font-size: medium; border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; ">
<br><table style="border-top:1px dashed rgb(204, 204, 204);border-bottom:1px dashed rgb(204, 204, 204);width:585px" cellpadding="5" width="100%"><tbody><tr><td><div style="font-family:'Lucida Grande',Verdana,Arial,sans-serif;font-size:9px;color:rgb(153, 153, 153);line-height:9pt">
<div style="float:left"><strong style="color:rgb(255, 0, 0)">Nicholas Tang</strong><strong style="color:rgb(85, 85, 85);margin-left:3px;margin-right:3px">:</strong></div><div style="float:left;font-weight:bold;color:rgb(85, 85, 85);margin-right:3px">
VP, Dev Ops<br></div><br><div style="float:left;margin-right:5px"><a href="mailto:nicholas.tang@livestream.com" title="Email me" style="color:rgb(153, 153, 153);text-decoration:none" target="_blank">nicholas.tang@livestream.com</a></div>
<div style="float:left;font-weight:bold;color:rgb(204, 204, 204);margin-right:5px">|</div><div style="float:left;margin-right:5px">t: +1 (646) 495 9707</div><div style="float:left;font-weight:bold;color:rgb(204, 204, 204);margin-right:5px">
|</div><div style="float:left;margin-right:5px">m: +1 (347) 410 6066<br></div><div style="float:left;font-weight:bold;color:rgb(204, 204, 204);margin-right:3px">|</div><div style="float:left">111 8th Avenue, Floor 15, New York, NY 10011</div>
</div></td><td style="width:100px" width="100px"><a href="http://www.livestream.com/" title="Visit livestream.com" target="_blank"><img src="http://livestream-email.s3.amazonaws.com/signatures/ls-logo-100x25.jpg" alt="www.livestream.com" border="0" height="25" width="100"></a></td>
</tr></tbody></table></div></div></span></div></span></div></span> </div><br>
</div>