From a.marinov at ucdn.com  Fri Oct  2 06:14:47 2015
From: a.marinov at ucdn.com (Anatoli Marinov)
Date: Fri, 2 Oct 2015 09:14:47 +0300
Subject: Slice module
In-Reply-To: <20150929160025.GC21123@Romans-MacBook-Air.local>
References: <20150929160025.GC21123@Romans-MacBook-Air.local>
Message-ID: <CA+HzRfRJ-4RSrkk8_wXPMB=evKON7=wChT35Zw3ncvYA6gtusQ@mail.gmail.com>

Thanks Roman! I tested it briefly and it looks good. Next days I think to
test it in limited production environment with high traffic load (~10Gb).

On Tue, Sep 29, 2015 at 7:00 PM, Roman Arutyunyan <arut at nginx.com> wrote:

> Hello,
>
> I'm happy to publish the experimental Slice module.  The module makes it
> possible to split a big upstream response into smaller parts and cache them
> independently.
>
> The module supports range requests.  When a part of a file is requested,
> only the required slice upstream requests are made.  If caching is enabled,
> future requests will only go to upstream for missing slices.
>
> The module adds
>
> - "slice" directive setting the slice size.
>
> - "$slice_range" variable, which must be added to the cache key
>   expression and passed to upstream as the Range header value.
>   The variable holds current slice range in the HTTP Range field format.
>
>
> Build
> -----
>
> Use the --with-http_slice_module configure script option.
>
>
> Example
> -------
>
> location / {
>     slice 1m;
>
>     proxy_cache cache;
>     proxy_cache_key $uri$is_args$args$slice_range;
>     proxy_set_header Range $slice_range;
>     proxy_cache_valid 200 206 1h;
>     proxy_pass http://127.0.0.1:9000;
> }
>
>
> Known issues
> ------------
>
> The module can lead to excessive memory and file handle usage.
>
>
> Thanks for testing.
>
> --
> Best wishes,
> Roman Arutyunyan
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151002/bd53cc96/attachment.html>

From dhananjaybalan at gmail.com  Fri Oct  2 17:20:58 2015
From: dhananjaybalan at gmail.com (Dhananjay Balan)
Date: Fri, 2 Oct 2015 22:50:58 +0530
Subject: Contributing to NGINX.
Message-ID: <CA+avMNJLVCbv573-weqbXn7W6f-sMueNy7-_W8_dOnYp-r5Q3Q@mail.gmail.com>

Hi,

I've been using NGINX in production for at least past three years.
Thanks for the project.

How does one start contributing code to NGINX?

I have seen the trac pages and I'm reading my way through the source
code. What I'm looking for is some issues/features that will help me
understand things little better? Or any other guidance is appreciated.

Thanks for reading so far. Apologies if I wasted your time.

- Dhananjay


From tfransosi at gmail.com  Mon Oct  5 00:21:53 2015
From: tfransosi at gmail.com (Thiago Farina)
Date: Sun, 4 Oct 2015 21:21:53 -0300
Subject: Contributing to NGINX.
In-Reply-To: <CA+avMNJLVCbv573-weqbXn7W6f-sMueNy7-_W8_dOnYp-r5Q3Q@mail.gmail.com>
References: <CA+avMNJLVCbv573-weqbXn7W6f-sMueNy7-_W8_dOnYp-r5Q3Q@mail.gmail.com>
Message-ID: <CACnwZYfRbmiUDdwTyrYF4hdCWUaZ4+G0X_t4dytdFWeFW62YOw@mail.gmail.com>

On Fri, Oct 2, 2015 at 2:20 PM, Dhananjay Balan
<dhananjaybalan at gmail.com> wrote:
> Hi,
>
> I've been using NGINX in production for at least past three years.
> Thanks for the project.
>
> How does one start contributing code to NGINX?
>
Have you encountered any bug while using it in production?

That may be a good start point for contributing to the project.

While using it, have you feel the lack of a feature? If yes, you might
propose implementing such new 'missing' feature to fill the gap.

Answering those questions might help you decide if investing time
contributing to the project is a good use of your time as well.

Best regards,

-- 
Thiago Farina


From a.marinov at ucdn.com  Mon Oct  5 14:30:25 2015
From: a.marinov at ucdn.com (Anatoli Marinov)
Date: Mon, 5 Oct 2015 17:30:25 +0300
Subject: How to remove argument from request line?
Message-ID: <CA+HzRfSWNp85SMfbhh9Eua8vBrgDy1JvR8vUDazLQNBi_KdREw@mail.gmail.com>

Hi,
I am writing a custom module for nginx which should remove particular
arguments from request line when request is sending to the origin through
upstream on cache miss. Proxy cache is used. In general I want to remove
start=XXX argument.
My handler is attached in rewrite phase and I tryed to modify r->args but
on my origin I received unmodified request string. I guess this is not the
right way.

Could you advise me how I can achieve this?

Thank in advance
Anatoli Marinov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151005/739859a1/attachment.html>

From doubleukay at doubleukay.com  Mon Oct  5 19:22:02 2015
From: doubleukay at doubleukay.com (Woon Wai Keen)
Date: Tue, 6 Oct 2015 03:22:02 +0800
Subject: Slice module
In-Reply-To: <20150929160025.GC21123@Romans-MacBook-Air.local>
References: <20150929160025.GC21123@Romans-MacBook-Air.local>
Message-ID: <5612CDDA.2070903@doubleukay.com>

On 30/9/2015 12:00 AM, Roman Arutyunyan wrote:
> Known issues
> ------------
>
> The module can lead to excessive memory and file handle usage.
Hi Roman, thanks for sharing. Under what circumstances will there be 
excessive memory and file handle usage?


From arut at nginx.com  Mon Oct  5 21:42:13 2015
From: arut at nginx.com (Roman Arutyunyan)
Date: Tue, 6 Oct 2015 00:42:13 +0300
Subject: Slice module
In-Reply-To: <5612CDDA.2070903@doubleukay.com>
References: <20150929160025.GC21123@Romans-MacBook-Air.local>
 <5612CDDA.2070903@doubleukay.com>
Message-ID: <E9ECE93D-FFDE-4A75-8334-6056D5C2B2A4@nginx.com>

Hello,

> On 05 Oct 2015, at 22:22, Woon Wai Keen <doubleukay at doubleukay.com> wrote:
> 
> On 30/9/2015 12:00 AM, Roman Arutyunyan wrote:
>> Known issues
>> ------------
>> 
>> The module can lead to excessive memory and file handle usage.
> Hi Roman, thanks for sharing. Under what circumstances will there be excessive memory and file handle usage?

Memory and file descriptors allocated by each subrequest are mostly
freed only when the user request ends.  If you have small slice size,
that will lead to many subrequests each allocating some resources.

--
Roman Arutyunyan




From agentzh at gmail.com  Tue Oct  6 03:01:54 2015
From: agentzh at gmail.com (Yichun Zhang (agentzh))
Date: Tue, 6 Oct 2015 11:01:54 +0800
Subject: subrequest error with http2
In-Reply-To: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1D8@PTPPICEX03.PTPortugal.corpPT.com>
References: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1D8@PTPPICEX03.PTPortugal.corpPT.com>
Message-ID: <CAB4Tn6OK+CN0FQ6yAcq814woxKgQ8WHgFOd9g91EBRteu83ARg@mail.gmail.com>

Hello!

On Wed, Sep 23, 2015 at 1:14 AM, Carlos Eduardo Ferreira Rodrigues wrote:
> I just upgraded nginx to 1.9.5 on our testing enviroment, and immediately started seeing this error on http2 requests:
>
> 2015/09/22 18:04:06 [alert] 27305#27305: *1 epoll_ctl(1, 17) failed (17: File exists), client: x.x.x.x, server: _, request: "GET ... HTTP/2.0", subrequest: "...", host: "..."
>
> The subrequest is being made from Lua code using "ngx.location.capture()", so I understand this may be an issue with ngx_lua and not nginx itself. However, this worked fine with nginx 1.8.0/1.9.4 and spdy and nothing else has changed.
>

The ngx.location.capture* API does not support SPDY nor HTTP 2.0
(yet). We may add such support in the future.

Oh, BTW, nginx 1.9.5 is not listed in the compatible nginx core list yet:

    https://github.com/openresty/lua-nginx-module#nginx-compatibility

Regards,
-agentzh


From agentzh at gmail.com  Tue Oct  6 03:04:27 2015
From: agentzh at gmail.com (Yichun Zhang (agentzh))
Date: Tue, 6 Oct 2015 11:04:27 +0800
Subject: subrequest error with http2
In-Reply-To: <20150922173440.GH13202@mdounin.ru>
References: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1D8@PTPPICEX03.PTPortugal.corpPT.com>
 <20150922173440.GH13202@mdounin.ru>
Message-ID: <CAB4Tn6Os_CQh+ugKj8hXwuGPcw8NZc845d7_u9KTVWAR46D+vQ@mail.gmail.com>

Hello!

On Wed, Sep 23, 2015 at 1:34 AM, Maxim Dounin wrote:
>
> The lua module deeply integrates into nginx internals (far beyond
> what we consider to be nginx modules API), and there is no surprise
> it's broken by the changes in nginx 1.9.5.
>

True. This is also why ngx_lua has so many advanced features :)
Actually I quite like the fact that NGINX does not enforce a limited
set of API for 3rd-party modules to use and 3rd-party module authors
can almost always find their way to implement bold imaginations *grin*

For the same reason, I'll hate to see that NGINX enforces a well
defined ABI, BTW :)

Regards,
-agentzh


From arut at nginx.com  Tue Oct  6 05:59:37 2015
From: arut at nginx.com (Roman Arutyunyan)
Date: Tue, 06 Oct 2015 05:59:37 +0000
Subject: [nginx] Stream: delete proxy connection timer after SSL handshake.
Message-ID: <hg.4b4aee40c508.1444111177.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/4b4aee40c508
branches:  
changeset: 6258:4b4aee40c508
user:      Ruslan Ermilov <ru at nginx.com>
date:      Tue Oct 06 08:57:09 2015 +0300
description:
Stream: delete proxy connection timer after SSL handshake.

The timer remained active and could drop active SSL connection.

diffstat:

 src/stream/ngx_stream_proxy_module.c |  4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diffs (14 lines):

diff -r 5eb4d7541107 -r 4b4aee40c508 src/stream/ngx_stream_proxy_module.c
--- a/src/stream/ngx_stream_proxy_module.c	Sun Jul 12 08:31:38 2015 -0700
+++ b/src/stream/ngx_stream_proxy_module.c	Tue Oct 06 08:57:09 2015 +0300
@@ -759,6 +759,10 @@ ngx_stream_proxy_ssl_handshake(ngx_conne
             u->peer.save_session(&u->peer, u->peer.data);
         }
 
+        if (pc->write->timer_set) {
+            ngx_del_timer(pc->write);
+        }
+
         ngx_stream_proxy_init_upstream(s);
 
         return;


From carlos-eduardo-rodrigues at telecom.pt  Tue Oct  6 09:33:12 2015
From: carlos-eduardo-rodrigues at telecom.pt (Carlos Eduardo Ferreira Rodrigues)
Date: Tue, 6 Oct 2015 10:33:12 +0100
Subject: subrequest error with http2
In-Reply-To: <CAB4Tn6OK+CN0FQ6yAcq814woxKgQ8WHgFOd9g91EBRteu83ARg@mail.gmail.com>
References: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1D8@PTPPICEX03.PTPortugal.corpPT.com>,
 <CAB4Tn6OK+CN0FQ6yAcq814woxKgQ8WHgFOd9g91EBRteu83ARg@mail.gmail.com>
Message-ID: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1EE@PTPPICEX03.PTPortugal.corpPT.com>

Hi,

I'm aware 1.9.5 isn't supported yet and that SPDY is mentioned in the docs as non-working for certain API calls as well. However, we are have been using location.capture with SPDY for a while now and haven't seen any issues. Information on why and what issues are present is a little fuzzy (and searching only returns old stuff that some sources claim not to apply anymore), can you provide some details?

Best regards,

--
Carlos Rodrigues

________________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] On Behalf Of Yichun Zhang (agentzh) [agentzh at gmail.com]
Sent: Tuesday, October 06, 2015 4:01
To: nginx-devel at nginx.org
Subject: Re: subrequest error with http2

Hello!

On Wed, Sep 23, 2015 at 1:14 AM, Carlos Eduardo Ferreira Rodrigues wrote:
> I just upgraded nginx to 1.9.5 on our testing enviroment, and immediately started seeing this error on http2 requests:
>
> 2015/09/22 18:04:06 [alert] 27305#27305: *1 epoll_ctl(1, 17) failed (17: File exists), client: x.x.x.x, server: _, request: "GET ... HTTP/2.0", subrequest: "...", host: "..."
>
> The subrequest is being made from Lua code using "ngx.location.capture()", so I understand this may be an issue with ngx_lua and not nginx itself. However, this worked fine with nginx 1.8.0/1.9.4 and spdy and nothing else has changed.
>

The ngx.location.capture* API does not support SPDY nor HTTP 2.0
(yet). We may add such support in the future.

Oh, BTW, nginx 1.9.5 is not listed in the compatible nginx core list yet:

    https://github.com/openresty/lua-nginx-module#nginx-compatibility

Regards,
-agentzh

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel


From devnexen at gmail.com  Wed Oct  7 04:53:45 2015
From: devnexen at gmail.com (David CARLIER)
Date: Wed, 7 Oct 2015 05:53:45 +0100
Subject: small feature
Message-ID: <CA+XhMqx5EnD-LCmc2REsP_UKXpdJ-acpcoOkWfVZiim9keDo3A@mail.gmail.com>

Hi all,

I developed a small feature which puts nginx in chroot mode, off by default
and when you set a prefix path via -p, although OpenBSD team did already a
version far before me.

Hope it is useful.

Kind regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151007/96515d1e/attachment.html>
-------------- next part --------------
diff -r 4b4aee40c508 src/core/nginx.c
--- a/src/core/nginx.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/core/nginx.c	Wed Oct 07 05:49:50 2015 +0100
@@ -49,6 +49,13 @@
       offsetof(ngx_core_conf_t, master),
       NULL },
 
+    { ngx_string("chroot"),
+      NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_FLAG,
+      ngx_conf_set_flag_slot,
+      0,
+      offsetof(ngx_core_conf_t, chroot),
+      NULL },
+
     { ngx_string("timer_resolution"),
       NGX_MAIN_CONF|NGX_DIRECT_CONF|NGX_CONF_TAKE1,
       ngx_conf_set_msec_slot,
@@ -967,6 +974,7 @@
 
     ccf->daemon = NGX_CONF_UNSET;
     ccf->master = NGX_CONF_UNSET;
+    ccf->chroot = NGX_CONF_UNSET;
     ccf->timer_resolution = NGX_CONF_UNSET_MSEC;
 
     ccf->worker_processes = NGX_CONF_UNSET;
@@ -984,6 +992,9 @@
         return NULL;
     }
 
+    ccf->chroot_directory.len = NGX_CONF_UNSET_UINT;
+    ccf->chroot_directory.data = NULL;
+
     return ccf;
 }
 
@@ -995,6 +1006,7 @@
 
     ngx_conf_init_value(ccf->daemon, 1);
     ngx_conf_init_value(ccf->master, 1);
+    ngx_conf_init_value(ccf->chroot, 0);
     ngx_conf_init_msec_value(ccf->timer_resolution, 0);
 
     ngx_conf_init_value(ccf->worker_processes, 1);
@@ -1014,7 +1026,6 @@
 
 #endif
 
-
     if (ccf->pid.len == 0) {
         ngx_str_set(&ccf->pid, NGX_PID_PATH);
     }
@@ -1063,6 +1074,19 @@
     }
 
 
+    if (ccf->chroot) {
+        if (ngx_prefix) {
+            ngx_uint_t ngx_prefixlen = ngx_strlen(ngx_prefix);
+            ccf->chroot_directory.len = ngx_prefixlen;
+            ccf->chroot_directory.data = ngx_palloc(cycle->pool, ngx_prefixlen + 1);
+            ngx_memcpy(ccf->chroot_directory.data, ngx_prefix, ngx_prefixlen);
+        } else if(ccf->chroot_directory.len == NGX_CONF_UNSET_UINT) {
+            ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
+                    "in chroot mode the prefix path must be set ");
+            return NGX_CONF_ERROR;
+        }
+    }
+
     if (ccf->lock_file.len == 0) {
         ngx_str_set(&ccf->lock_file, NGX_LOCK_PATH);
     }
diff -r 4b4aee40c508 src/core/ngx_cycle.c
--- a/src/core/ngx_cycle.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/core/ngx_cycle.c	Wed Oct 07 05:49:50 2015 +0100
@@ -1087,6 +1087,8 @@
     ngx_uint_t        i;
     ngx_list_part_t  *part;
     ngx_open_file_t  *file;
+    ngx_core_conf_t  *ccf = (ngx_core_conf_t *)ngx_get_conf(cycle->conf_ctx,
+            ngx_core_module);
 
     part = &cycle->open_files.part;
     file = part->elts;
@@ -1106,6 +1108,8 @@
             continue;
         }
 
+        ngx_adjust_path(ccf, cycle, &file[i].name);
+
         if (file[i].flush) {
             file[i].flush(&file[i], cycle->log);
         }
@@ -1278,6 +1282,34 @@
     return shm_zone;
 }
 
+void
+ngx_adjust_path(ngx_core_conf_t *ccf, ngx_cycle_t *cycle, ngx_str_t *path)
+{
+    if (!ccf->chroot)
+        return;
+
+    ngx_uint_t cpathlen = path->len;
+    u_char *cpath = path->data;
+    cpath[cpathlen] = '\0';
+
+    if (cycle->prefix.data != NULL) {
+        u_char *prefix = cycle->prefix.data;
+        prefix[cycle->prefix.len] = '\0';
+
+        if (ngx_strnstr(cpath, prefix, cycle->prefix.len)) {
+            cpath += cycle->prefix.len;
+            cpathlen -= cycle->prefix.len;
+        }
+    }
+
+    if (cpath[0] != '/') {
+        u_char *newpath = ngx_palloc(cycle->pool, cpathlen + 2);
+        newpath[0] = '/';
+        ngx_memcpy(newpath + 1, cpath, cpathlen);
+        path->data = newpath;
+        path->len = cpathlen + 1;
+    }
+}
 
 static void
 ngx_clean_old_cycles(ngx_event_t *ev)
diff -r 4b4aee40c508 src/core/ngx_cycle.h
--- a/src/core/ngx_cycle.h	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/core/ngx_cycle.h	Wed Oct 07 05:49:50 2015 +0100
@@ -77,6 +77,7 @@
 typedef struct {
      ngx_flag_t               daemon;
      ngx_flag_t               master;
+     ngx_flag_t               chroot;
 
      ngx_msec_t               timer_resolution;
 
@@ -97,6 +98,7 @@
 
      ngx_str_t                working_directory;
      ngx_str_t                lock_file;
+     ngx_str_t				  chroot_directory;
 
      ngx_str_t                pid;
      ngx_str_t                oldpid;
@@ -119,6 +121,7 @@
 uint64_t ngx_get_cpu_affinity(ngx_uint_t n);
 ngx_shm_zone_t *ngx_shared_memory_add(ngx_conf_t *cf, ngx_str_t *name,
     size_t size, void *tag);
+void ngx_adjust_path(ngx_core_conf_t *, ngx_cycle_t *, ngx_str_t *);
 
 
 extern volatile ngx_cycle_t  *ngx_cycle;
diff -r 4b4aee40c508 src/core/ngx_file.c
--- a/src/core/ngx_file.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/core/ngx_file.c	Wed Oct 07 05:49:50 2015 +0100
@@ -566,9 +566,12 @@
     ngx_err_t         err;
     ngx_uint_t        i;
     ngx_path_t      **path;
+    ngx_core_conf_t *ccf = (ngx_core_conf_t *)ngx_get_conf(cycle->conf_ctx,
+            ngx_core_module);
 
     path = cycle->paths.elts;
     for (i = 0; i < cycle->paths.nelts; i++) {
+        ngx_adjust_path(ccf, cycle, &path[i]->name);
 
         if (ngx_create_dir(path[i]->name.data, 0700) == NGX_FILE_ERROR) {
             err = ngx_errno;
diff -r 4b4aee40c508 src/http/modules/ngx_http_auth_basic_module.c
--- a/src/http/modules/ngx_http_auth_basic_module.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/http/modules/ngx_http_auth_basic_module.c	Wed Oct 07 05:49:50 2015 +0100
@@ -441,6 +441,8 @@
 ngx_http_auth_basic_user_file(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_http_auth_basic_loc_conf_t *alcf = conf;
+    ngx_core_conf_t				   *ccf =
+        (ngx_core_conf_t *)ngx_get_conf(cf->cycle->conf_ctx, ngx_core_module);
 
     ngx_str_t                         *value;
     ngx_http_compile_complex_value_t   ccv;
@@ -453,6 +455,8 @@
 
     ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t));
 
+    ngx_adjust_path(ccf, cf->cycle, &value[1]);
+
     ccv.cf = cf;
     ccv.value = &value[1];
     ccv.complex_value = &alcf->user_file;
diff -r 4b4aee40c508 src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/http/ngx_http_core_module.c	Wed Oct 07 05:49:50 2015 +0100
@@ -4430,6 +4430,8 @@
 ngx_http_core_root(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
 {
     ngx_http_core_loc_conf_t *clcf = conf;
+    ngx_core_conf_t          *ccf =
+        (ngx_core_conf_t *)ngx_get_conf(cf->cycle->conf_ctx, ngx_core_module);
 
     ngx_str_t                  *value;
     ngx_int_t                   alias;
@@ -4486,9 +4488,12 @@
         return NGX_CONF_ERROR;
     }
 
+    ngx_adjust_path(ccf, cf->cycle, &value[1]);
+
     clcf->alias = alias ? clcf->name.len : 0;
     clcf->root = value[1];
 
+
     if (!alias && clcf->root.data[clcf->root.len - 1] == '/') {
         clcf->root.len--;
     }
diff -r 4b4aee40c508 src/os/unix/ngx_process_cycle.c
--- a/src/os/unix/ngx_process_cycle.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/os/unix/ngx_process_cycle.c	Wed Oct 07 05:49:50 2015 +0100
@@ -828,6 +828,28 @@
     }
 
     if (geteuid() == 0) {
+        if (ccf->chroot) {
+            u_char *chroot_directory = ccf->chroot_directory.data;
+            chroot_directory[ccf->chroot_directory.len] = '\0';
+
+            if (chroot((const char *)chroot_directory) == -1) {
+                ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
+                        "chroot(%s) failed", chroot_directory);
+                /* fatal */
+                exit(2);
+            }
+
+            if (chdir("/") == -1) {
+                ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
+                        "chdir failed");
+                /* fatal */
+                exit(2);
+            }
+
+            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "chrooted into %s", chroot_directory);
+
+        }
+
         if (setgid(ccf->group) == -1) {
             ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                           "setgid(%d) failed", ccf->group);

From agentzh at gmail.com  Wed Oct  7 09:42:53 2015
From: agentzh at gmail.com (Yichun Zhang (agentzh))
Date: Wed, 7 Oct 2015 17:42:53 +0800
Subject: subrequest error with http2
In-Reply-To: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1EE@PTPPICEX03.PTPortugal.corpPT.com>
References: <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1D8@PTPPICEX03.PTPortugal.corpPT.com>
 <CAB4Tn6OK+CN0FQ6yAcq814woxKgQ8WHgFOd9g91EBRteu83ARg@mail.gmail.com>
 <CD4E28A8D5DAB64E83C2C6750806D2F7B049F6A1EE@PTPPICEX03.PTPortugal.corpPT.com>
Message-ID: <CAB4Tn6NLnhtyFS6L+_EX8fS7BdErcZ8okB3Qgp5HykVtFMJm=g@mail.gmail.com>

Hello!

On Tue, Oct 6, 2015 at 5:33 PM, Carlos Eduardo Ferreira Rodrigues wrote:
> I'm aware 1.9.5 isn't supported yet and that SPDY is mentioned in the docs as non-working for certain API calls
> as well. However, we are have been using location.capture with SPDY for a while now and haven't seen any issues.
> Information on why and what issues are present is a little fuzzy (and searching only returns old stuff that some
> sources claim not to apply anymore), can you provide some details?
>

Well, I just haven't had the time to look close enough to SPDY or
HTTP/2. I've been mainly relying on the user feedback on such
(unsupported) usage. In theory, they *should* work with
ngx.location.capture*. But there might be some knobs to resolve. I'm
not sure and I cannot give you any details at this point. Sorry.

Regards,
-agentzh


From mdounin at mdounin.ru  Wed Oct  7 15:13:15 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 7 Oct 2015 18:13:15 +0300
Subject: small feature
In-Reply-To: <CA+XhMqx5EnD-LCmc2REsP_UKXpdJ-acpcoOkWfVZiim9keDo3A@mail.gmail.com>
References: <CA+XhMqx5EnD-LCmc2REsP_UKXpdJ-acpcoOkWfVZiim9keDo3A@mail.gmail.com>
Message-ID: <20151007151315.GL30105@mdounin.ru>

Hello!

On Wed, Oct 07, 2015 at 05:53:45AM +0100, David CARLIER wrote:

> Hi all,
> 
> I developed a small feature which puts nginx in chroot mode, off by default
> and when you set a prefix path via -p, although OpenBSD team did already a
> version far before me.

I believe I've already commented mostly the same patch in trac 
ticket #805 (https://trac.nginx.org/nginx/ticket/805):

: Chrooting to the prefix looks like a wrong idea, in particular, 
: because many paths, including compiled in ones, can be outside of 
: the prefix. I would rather think of chroot()'ing to some arbitrary 
: path. The part trying to do adjustments in ngx_http_core_root() 
: looks very wrong, too, you shouldn't try to adjust anything agains 
: chroot() path at that level.

The main problem with chroot() is how to address path changes 
between master and worker processes.  And it needs some solution 
transparent to various modules.  Otherwise, it would be a real 
pain to support chroot() here and there.

I don't really see the problem addressed in the patch.  If you 
think it is, please provide details on how it's expected to work.

Some more comments below.  They are probably useless per se, as I 
think the code should be seriously rewritten to address the above 
comments, but may help you if you are planning to work further on 
nginx code.

[...]

> @@ -1014,7 +1026,6 @@

Please use

[diff]
showfunc=1

in the Mercurial configuration, it greatly simplifies reading 
patches.

>  
>  #endif
>  
> -
>      if (ccf->pid.len == 0) {
>          ngx_str_set(&ccf->pid, NGX_PID_PATH);
>      }

Please avoid unrelated (and incorrect) style changes.

> @@ -1063,6 +1074,19 @@
>      }
>  
>  
> +    if (ccf->chroot) {
> +        if (ngx_prefix) {
> +            ngx_uint_t ngx_prefixlen = ngx_strlen(ngx_prefix);

Style: variables should be defined at function start, not in the 
body; variables initialization shouldn't be done in declaration.

> +            ccf->chroot_directory.len = ngx_prefixlen;
> +            ccf->chroot_directory.data = ngx_palloc(cycle->pool, ngx_prefixlen + 1);
> +            ngx_memcpy(ccf->chroot_directory.data, ngx_prefix, ngx_prefixlen);
> +        } else if(ccf->chroot_directory.len == NGX_CONF_UNSET_UINT) {

Style: there should be a space between "if" and "(".

> +            ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
> +                    "in chroot mode the prefix path must be set ");

Style: incorrect indentation; extra space at the end of the 
message.  Further style comments suppressed, there are lots of 
style issues.

It's not clear why do you require -p for chroot.  The -p switch is 
just one of many ways to set prefix, it shouldn't be something 
specific.

It's not clear why do you test ccf->chroot_directory.len, as it's 
only set in the "if (ngx_prefix)" part above.  That is, the test 
is useless.

[...]

> +    if (cycle->prefix.data != NULL) {
> +        u_char *prefix = cycle->prefix.data;
> +        prefix[cycle->prefix.len] = '\0';

Adding NUL character here either useless (if prefix is already 
NUL-terminated), or incorrect (what guarantees that there is a 
space for NUL?).

[...]

-- 
Maxim Dounin
http://nginx.org/


From vl at nginx.com  Thu Oct  8 13:05:15 2015
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 08 Oct 2015 13:05:15 +0000
Subject: [nginx] SSL: handled long string truncation in ngx_ssl_error().
Message-ID: <hg.2f34ea503ac4.1444309515.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/2f34ea503ac4
branches:  
changeset: 6259:2f34ea503ac4
user:      Vladimir Homutov <vl at nginx.com>
date:      Wed Oct 07 22:19:42 2015 +0300
description:
SSL: handled long string truncation in ngx_ssl_error().

If no space left in buffer after adding formatting symbols, error message
could be left without terminating null.  The fix is to output message using
actual length.

diffstat:

 src/event/ngx_event_openssl.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 4b4aee40c508 -r 2f34ea503ac4 src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c	Tue Oct 06 08:57:09 2015 +0300
+++ b/src/event/ngx_event_openssl.c	Wed Oct 07 22:19:42 2015 +0300
@@ -2045,7 +2045,7 @@ ngx_ssl_error(ngx_uint_t level, ngx_log_
         (void) ERR_get_error();
     }
 
-    ngx_log_error(level, log, err, "%s)", errstr);
+    ngx_log_error(level, log, err, "%*s)", p - errstr, errstr);
 }
 
 


From adontz at gmail.com  Sun Oct 11 20:20:32 2015
From: adontz at gmail.com (Roman Akopov)
Date: Mon, 12 Oct 2015 00:20:32 +0400
Subject: nginScript feedback
Message-ID: <004f01d10462$4890f2a0$d9b2d7e0$@gmail.com>

Hello,

 

I've just reviewed nginScript and saw you want feedback to be posted in this
mailing list, so here I am.

 

But, before posting any specific feedback, I'd better get to know your
vision of this project. For instance, is nginScript planned to serve NGINX
needs only (like JavaScriptCore is for WebKit only), or is it more generic
"embeddable JS engine" project with some unusual design decisions (like not
implementing some language constructs) targeted on non-browser audience?

 

Regards,

Roman

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151012/14823589/attachment.html>

From mhenrion at appnexus.com  Mon Oct 12 14:12:47 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Mon, 12 Oct 2015 14:12:47 +0000
Subject: Setting a custom header on a proxied request
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>

Hi all,


I'm currently writing an nginx module that sends a number of subrequests to upstream servers based on application logic, and then aggregates the results and send them back. That part currently works just fine, thanks to the help I was given here in the past.

However, I am now trying to add a custom header to the request that is being proxied and currently failing at it. For various technical reasons, I need to add a header that contains the IP address and port (from the point of view of nginx) of the upstream server currently being contacted. In order to get at that address, I have hooked my module into the peer selection mechanism, much like the keepalive, ip_hash modules and others do. So far so good, I get at that address and can log it just fine. However, I can't seem to be able to add it as a header to my request.

Here's my init_peer callback :

static ngx_int_t
ngx_http_dispatcher_upstream_init_peer(ngx_http_request_t *r,
    ngx_http_upstream_srv_conf_t *us)
{
        ngx_http_dispatcher_upstream_peer_data_t *dp;
        ngx_http_dispatcher_srv_conf_t *dscf;
        ngx_int_t rc;

        dscf = ngx_http_conf_upstream_srv_conf(us, ngx_http_dispatcher_module);

        dp = ngx_palloc(r->pool, sizeof(ngx_http_dispatcher_upstream_peer_data_t));
        if (dp == NULL) {
                return NGX_ERROR;
        }

        rc = dscf->original_init_peer(r, us);
        if (rc != NGX_OK) {
                return rc;
        }

        dp->request = r;
        dp->data = r->upstream->peer.data;
        dp->original_get_peer = r->upstream->peer.get;
        dp->original_free_peer = r->upstream->peer.free;

        r->upstream->peer.data = dp;
        r->upstream->peer.get = ngx_http_dispatcher_upstream_get_peer;
        r->upstream->peer.free = ngx_http_dispatcher_upstream_free_peer;

#if (NGX_HTTP_SSL)
        dp->original_set_session = r->upstream->peer.set_session;
        dp->original_save_session = r->upstream->peer.save_session;
        r->upstream->peer.set_session = ngx_http_dispatcher_upstream_set_session;
        r->upstream->peer.save_session = ngx_http_dispatcher_upstream_save_session;
#endif

        return NGX_OK;
}

Note that I save the pointer to the ngx_http_request_t object in my peer data structure to access it later. Now here's my peer selection hook :

static ngx_int_t
ngx_http_dispatcher_upstream_get_peer(ngx_peer_connection_t *pc, void *data)
{
        ngx_http_dispatcher_upstream_peer_data_t *dp = data;
        u_char addrbuf[NGX_SOCKADDR_STRLEN];
        ngx_http_request_t *r;
        ngx_table_elt_t *h;
        ngx_str_t peeraddr;
        ngx_int_t rc;

        rc = dp->original_get_peer(pc, dp->data);
        if (rc != NGX_OK) {
                return rc;
        }

        peeraddr.data = addrbuf;
        peeraddr.len = ngx_sock_ntop(pc->sockaddr, pc->socklen, addrbuf,
            sizeof(addrbuf), 1);

        ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0,
            "Selected peer address is: %V", &peeraddr);

        r = dp->request;

        h = ngx_list_push(&r->headers_in.headers);
        if (h == NULL) {
                return NGX_ERROR;
        }
        h->hash = 1;
        ngx_str_set(&h->key, "X-Shard-Addr");
        h->value.data = ngx_pstrdup(r->pool, &peeraddr);
        if (h->value.data == NULL) {
                return NGX_ERROR;
        }

        h->value.len = peeraddr.len;

        return NGX_OK;
}

So after getting at the actual address we're forwarding our request to, I try to add a custom header to r->headers_in, on the request object that I saved in my peer data.

I suspect I'm not actually dealing with the correct request here, because no matter what I try, I can't get this code to actually alter the headers of the forwarded request in any way.

Any help would be greatly appreciated!
Thanks,
Maxime

From ahutchings at nginx.com  Mon Oct 12 16:04:18 2015
From: ahutchings at nginx.com (Andrew Hutchings)
Date: Mon, 12 Oct 2015 17:04:18 +0100
Subject: Setting a custom header on a proxied request
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <561BDA02.9090908@nginx.com>

Hi Maxime,

I could be wrong here (I'm not an expert at this section of code), but I
believe what you need to set is:

r->upstream->headers_in

If this doesn't work please let me know and I'll build a test to try it
out here.

Kind Regards
Andrew

On 12/10/15 15:12, Maxime Henrion wrote:
> Hi all,
> 
> 
> I'm currently writing an nginx module that sends a number of subrequests to upstream servers based on application logic, and then aggregates the results and send them back. That part currently works just fine, thanks to the help I was given here in the past.
> 
> However, I am now trying to add a custom header to the request that is being proxied and currently failing at it. For various technical reasons, I need to add a header that contains the IP address and port (from the point of view of nginx) of the upstream server currently being contacted. In order to get at that address, I have hooked my module into the peer selection mechanism, much like the keepalive, ip_hash modules and others do. So far so good, I get at that address and can log it just fine. However, I can't seem to be able to add it as a header to my request.
> 
> Here's my init_peer callback :
> 
> static ngx_int_t
> ngx_http_dispatcher_upstream_init_peer(ngx_http_request_t *r,
>     ngx_http_upstream_srv_conf_t *us)
> {
>         ngx_http_dispatcher_upstream_peer_data_t *dp;
>         ngx_http_dispatcher_srv_conf_t *dscf;
>         ngx_int_t rc;
> 
>         dscf = ngx_http_conf_upstream_srv_conf(us, ngx_http_dispatcher_module);
> 
>         dp = ngx_palloc(r->pool, sizeof(ngx_http_dispatcher_upstream_peer_data_t));
>         if (dp == NULL) {
>                 return NGX_ERROR;
>         }
> 
>         rc = dscf->original_init_peer(r, us);
>         if (rc != NGX_OK) {
>                 return rc;
>         }
> 
>         dp->request = r;
>         dp->data = r->upstream->peer.data;
>         dp->original_get_peer = r->upstream->peer.get;
>         dp->original_free_peer = r->upstream->peer.free;
> 
>         r->upstream->peer.data = dp;
>         r->upstream->peer.get = ngx_http_dispatcher_upstream_get_peer;
>         r->upstream->peer.free = ngx_http_dispatcher_upstream_free_peer;
> 
> #if (NGX_HTTP_SSL)
>         dp->original_set_session = r->upstream->peer.set_session;
>         dp->original_save_session = r->upstream->peer.save_session;
>         r->upstream->peer.set_session = ngx_http_dispatcher_upstream_set_session;
>         r->upstream->peer.save_session = ngx_http_dispatcher_upstream_save_session;
> #endif
> 
>         return NGX_OK;
> }
> 
> Note that I save the pointer to the ngx_http_request_t object in my peer data structure to access it later. Now here's my peer selection hook :
> 
> static ngx_int_t
> ngx_http_dispatcher_upstream_get_peer(ngx_peer_connection_t *pc, void *data)
> {
>         ngx_http_dispatcher_upstream_peer_data_t *dp = data;
>         u_char addrbuf[NGX_SOCKADDR_STRLEN];
>         ngx_http_request_t *r;
>         ngx_table_elt_t *h;
>         ngx_str_t peeraddr;
>         ngx_int_t rc;
> 
>         rc = dp->original_get_peer(pc, dp->data);
>         if (rc != NGX_OK) {
>                 return rc;
>         }
> 
>         peeraddr.data = addrbuf;
>         peeraddr.len = ngx_sock_ntop(pc->sockaddr, pc->socklen, addrbuf,
>             sizeof(addrbuf), 1);
> 
>         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0,
>             "Selected peer address is: %V", &peeraddr);
> 
>         r = dp->request;
> 
>         h = ngx_list_push(&r->headers_in.headers);
>         if (h == NULL) {
>                 return NGX_ERROR;
>         }
>         h->hash = 1;
>         ngx_str_set(&h->key, "X-Shard-Addr");
>         h->value.data = ngx_pstrdup(r->pool, &peeraddr);
>         if (h->value.data == NULL) {
>                 return NGX_ERROR;
>         }
> 
>         h->value.len = peeraddr.len;
> 
>         return NGX_OK;
> }
> 
> So after getting at the actual address we're forwarding our request to, I try to add a custom header to r->headers_in, on the request object that I saved in my peer data.
> 
> I suspect I'm not actually dealing with the correct request here, because no matter what I try, I can't get this code to actually alter the headers of the forwarded request in any way.
> 
> Any help would be greatly appreciated!
> Thanks,
> Maxime
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
> 

-- 
Andrew Hutchings (LinuxJedi)
Senior Developer Advocate, Nginx Inc.


From mhenrion at appnexus.com  Mon Oct 12 17:04:03 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Mon, 12 Oct 2015 17:04:03 +0000
Subject: Setting a custom header on a proxied request
In-Reply-To: <561BDA02.9090908@nginx.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>,
 <561BDA02.9090908@nginx.com>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A5D194B80@01-exch-nym2.ad.corp.appnexus.com>

Hello Andrew,

Thank you for your answer!

I tried setting my header in r->upstream->headers_in but unfortunately, it seems the headers ngx_list_t isn't initialized yet at this point, and thus I end up with a crash in ngx_list_push(). I wish I could just initialize it myself, but it would most likely get overwritten when the actual initialization takes place.

Thanks,
Maxime
________________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Andrew Hutchings [ahutchings at nginx.com]
Sent: Monday, October 12, 2015 6:04 PM
To: nginx-devel at nginx.org
Subject: Re: Setting a custom header on a proxied request

Hi Maxime,

I could be wrong here (I'm not an expert at this section of code), but I
believe what you need to set is:

r->upstream->headers_in

If this doesn't work please let me know and I'll build a test to try it
out here.

Kind Regards
Andrew

On 12/10/15 15:12, Maxime Henrion wrote:
> Hi all,
>
>
> I'm currently writing an nginx module that sends a number of subrequests to upstream servers based on application logic, and then aggregates the results and send them back. That part currently works just fine, thanks to the help I was given here in the past.
>
> However, I am now trying to add a custom header to the request that is being proxied and currently failing at it. For various technical reasons, I need to add a header that contains the IP address and port (from the point of view of nginx) of the upstream server currently being contacted. In order to get at that address, I have hooked my module into the peer selection mechanism, much like the keepalive, ip_hash modules and others do. So far so good, I get at that address and can log it just fine. However, I can't seem to be able to add it as a header to my request.
>
> Here's my init_peer callback :
>
> static ngx_int_t
> ngx_http_dispatcher_upstream_init_peer(ngx_http_request_t *r,
>     ngx_http_upstream_srv_conf_t *us)
> {
>         ngx_http_dispatcher_upstream_peer_data_t *dp;
>         ngx_http_dispatcher_srv_conf_t *dscf;
>         ngx_int_t rc;
>
>         dscf = ngx_http_conf_upstream_srv_conf(us, ngx_http_dispatcher_module);
>
>         dp = ngx_palloc(r->pool, sizeof(ngx_http_dispatcher_upstream_peer_data_t));
>         if (dp == NULL) {
>                 return NGX_ERROR;
>         }
>
>         rc = dscf->original_init_peer(r, us);
>         if (rc != NGX_OK) {
>                 return rc;
>         }
>
>         dp->request = r;
>         dp->data = r->upstream->peer.data;
>         dp->original_get_peer = r->upstream->peer.get;
>         dp->original_free_peer = r->upstream->peer.free;
>
>         r->upstream->peer.data = dp;
>         r->upstream->peer.get = ngx_http_dispatcher_upstream_get_peer;
>         r->upstream->peer.free = ngx_http_dispatcher_upstream_free_peer;
>
> #if (NGX_HTTP_SSL)
>         dp->original_set_session = r->upstream->peer.set_session;
>         dp->original_save_session = r->upstream->peer.save_session;
>         r->upstream->peer.set_session = ngx_http_dispatcher_upstream_set_session;
>         r->upstream->peer.save_session = ngx_http_dispatcher_upstream_save_session;
> #endif
>
>         return NGX_OK;
> }
>
> Note that I save the pointer to the ngx_http_request_t object in my peer data structure to access it later. Now here's my peer selection hook :
>
> static ngx_int_t
> ngx_http_dispatcher_upstream_get_peer(ngx_peer_connection_t *pc, void *data)
> {
>         ngx_http_dispatcher_upstream_peer_data_t *dp = data;
>         u_char addrbuf[NGX_SOCKADDR_STRLEN];
>         ngx_http_request_t *r;
>         ngx_table_elt_t *h;
>         ngx_str_t peeraddr;
>         ngx_int_t rc;
>
>         rc = dp->original_get_peer(pc, dp->data);
>         if (rc != NGX_OK) {
>                 return rc;
>         }
>
>         peeraddr.data = addrbuf;
>         peeraddr.len = ngx_sock_ntop(pc->sockaddr, pc->socklen, addrbuf,
>             sizeof(addrbuf), 1);
>
>         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0,
>             "Selected peer address is: %V", &peeraddr);
>
>         r = dp->request;
>
>         h = ngx_list_push(&r->headers_in.headers);
>         if (h == NULL) {
>                 return NGX_ERROR;
>         }
>         h->hash = 1;
>         ngx_str_set(&h->key, "X-Shard-Addr");
>         h->value.data = ngx_pstrdup(r->pool, &peeraddr);
>         if (h->value.data == NULL) {
>                 return NGX_ERROR;
>         }
>
>         h->value.len = peeraddr.len;
>
>         return NGX_OK;
> }
>
> So after getting at the actual address we're forwarding our request to, I try to add a custom header to r->headers_in, on the request object that I saved in my peer data.
>
> I suspect I'm not actually dealing with the correct request here, because no matter what I try, I can't get this code to actually alter the headers of the forwarded request in any way.
>
> Any help would be greatly appreciated!
> Thanks,
> Maxime
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>

--
Andrew Hutchings (LinuxJedi)
Senior Developer Advocate, Nginx Inc.

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel


From neale at sinenomine.net  Mon Oct 12 17:23:04 2015
From: neale at sinenomine.net (Neale Ferguson)
Date: Mon, 12 Oct 2015 17:23:04 +0000
Subject: Patch for s390x support
Message-ID: <D24164B4.44E00%neale@sinenomine.net>

Hi,
I would like to contribute the a fix to enable the Linux s390x platform.
The fix was built against today?s mercurial master and pertains to the gcc
atomic functions. I note the other architectures use inline assembler
rather than the gcc builtin operations. Is this for historical reasons?

Neale


From sorin.v.manole at gmail.com  Mon Oct 12 18:07:07 2015
From: sorin.v.manole at gmail.com (Sorin Manole)
Date: Mon, 12 Oct 2015 21:07:07 +0300
Subject: Setting a custom header on a proxied request
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A5D194B80@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
 <561BDA02.9090908@nginx.com>
 <EB010ACFBA76EF42ADCF5E994F19747A5D194B80@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <CAH19K_uEX7NQM_TPGiu9+t7CMA6tjOcerAh++5RXD1n8WgYOLw@mail.gmail.com>

Hello,
This doesn't work probably because the request data (headers + body) that
is sent to the upstream is "compiled" before the actual connection is
established.
See http://lxr.nginx.org/source/src/http/ngx_http_upstream.c#0574
->create_request(...) is called before ngx_http_upstream_connect(...)

2015-10-12 20:04 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com>:

> Hello Andrew,
>
> Thank you for your answer!
>
> I tried setting my header in r->upstream->headers_in but unfortunately, it
> seems the headers ngx_list_t isn't initialized yet at this point, and thus
> I end up with a crash in ngx_list_push(). I wish I could just initialize it
> myself, but it would most likely get overwritten when the actual
> initialization takes place.
>
> Thanks,
> Maxime
> ________________________________________
> From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Andrew
> Hutchings [ahutchings at nginx.com]
> Sent: Monday, October 12, 2015 6:04 PM
> To: nginx-devel at nginx.org
> Subject: Re: Setting a custom header on a proxied request
>
> Hi Maxime,
>
> I could be wrong here (I'm not an expert at this section of code), but I
> believe what you need to set is:
>
> r->upstream->headers_in
>
> If this doesn't work please let me know and I'll build a test to try it
> out here.
>
> Kind Regards
> Andrew
>
> On 12/10/15 15:12, Maxime Henrion wrote:
> > Hi all,
> >
> >
> > I'm currently writing an nginx module that sends a number of subrequests
> to upstream servers based on application logic, and then aggregates the
> results and send them back. That part currently works just fine, thanks to
> the help I was given here in the past.
> >
> > However, I am now trying to add a custom header to the request that is
> being proxied and currently failing at it. For various technical reasons, I
> need to add a header that contains the IP address and port (from the point
> of view of nginx) of the upstream server currently being contacted. In
> order to get at that address, I have hooked my module into the peer
> selection mechanism, much like the keepalive, ip_hash modules and others
> do. So far so good, I get at that address and can log it just fine.
> However, I can't seem to be able to add it as a header to my request.
> >
> > Here's my init_peer callback :
> >
> > static ngx_int_t
> > ngx_http_dispatcher_upstream_init_peer(ngx_http_request_t *r,
> >     ngx_http_upstream_srv_conf_t *us)
> > {
> >         ngx_http_dispatcher_upstream_peer_data_t *dp;
> >         ngx_http_dispatcher_srv_conf_t *dscf;
> >         ngx_int_t rc;
> >
> >         dscf = ngx_http_conf_upstream_srv_conf(us,
> ngx_http_dispatcher_module);
> >
> >         dp = ngx_palloc(r->pool,
> sizeof(ngx_http_dispatcher_upstream_peer_data_t));
> >         if (dp == NULL) {
> >                 return NGX_ERROR;
> >         }
> >
> >         rc = dscf->original_init_peer(r, us);
> >         if (rc != NGX_OK) {
> >                 return rc;
> >         }
> >
> >         dp->request = r;
> >         dp->data = r->upstream->peer.data;
> >         dp->original_get_peer = r->upstream->peer.get;
> >         dp->original_free_peer = r->upstream->peer.free;
> >
> >         r->upstream->peer.data = dp;
> >         r->upstream->peer.get = ngx_http_dispatcher_upstream_get_peer;
> >         r->upstream->peer.free = ngx_http_dispatcher_upstream_free_peer;
> >
> > #if (NGX_HTTP_SSL)
> >         dp->original_set_session = r->upstream->peer.set_session;
> >         dp->original_save_session = r->upstream->peer.save_session;
> >         r->upstream->peer.set_session =
> ngx_http_dispatcher_upstream_set_session;
> >         r->upstream->peer.save_session =
> ngx_http_dispatcher_upstream_save_session;
> > #endif
> >
> >         return NGX_OK;
> > }
> >
> > Note that I save the pointer to the ngx_http_request_t object in my peer
> data structure to access it later. Now here's my peer selection hook :
> >
> > static ngx_int_t
> > ngx_http_dispatcher_upstream_get_peer(ngx_peer_connection_t *pc, void
> *data)
> > {
> >         ngx_http_dispatcher_upstream_peer_data_t *dp = data;
> >         u_char addrbuf[NGX_SOCKADDR_STRLEN];
> >         ngx_http_request_t *r;
> >         ngx_table_elt_t *h;
> >         ngx_str_t peeraddr;
> >         ngx_int_t rc;
> >
> >         rc = dp->original_get_peer(pc, dp->data);
> >         if (rc != NGX_OK) {
> >                 return rc;
> >         }
> >
> >         peeraddr.data = addrbuf;
> >         peeraddr.len = ngx_sock_ntop(pc->sockaddr, pc->socklen, addrbuf,
> >             sizeof(addrbuf), 1);
> >
> >         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, pc->log, 0,
> >             "Selected peer address is: %V", &peeraddr);
> >
> >         r = dp->request;
> >
> >         h = ngx_list_push(&r->headers_in.headers);
> >         if (h == NULL) {
> >                 return NGX_ERROR;
> >         }
> >         h->hash = 1;
> >         ngx_str_set(&h->key, "X-Shard-Addr");
> >         h->value.data = ngx_pstrdup(r->pool, &peeraddr);
> >         if (h->value.data == NULL) {
> >                 return NGX_ERROR;
> >         }
> >
> >         h->value.len = peeraddr.len;
> >
> >         return NGX_OK;
> > }
> >
> > So after getting at the actual address we're forwarding our request to,
> I try to add a custom header to r->headers_in, on the request object that I
> saved in my peer data.
> >
> > I suspect I'm not actually dealing with the correct request here,
> because no matter what I try, I can't get this code to actually alter the
> headers of the forwarded request in any way.
> >
> > Any help would be greatly appreciated!
> > Thanks,
> > Maxime
> > _______________________________________________
> > nginx-devel mailing list
> > nginx-devel at nginx.org
> > http://mailman.nginx.org/mailman/listinfo/nginx-devel
> >
>
> --
> Andrew Hutchings (LinuxJedi)
> Senior Developer Advocate, Nginx Inc.
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151012/aa3d4db6/attachment.html>

From mdounin at mdounin.ru  Tue Oct 13 12:44:08 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 13 Oct 2015 15:44:08 +0300
Subject: Patch for s390x support
In-Reply-To: <D24164B4.44E00%neale@sinenomine.net>
References: <D24164B4.44E00%neale@sinenomine.net>
Message-ID: <20151013124408.GY30105@mdounin.ru>

Hello!

On Mon, Oct 12, 2015 at 05:23:04PM +0000, Neale Ferguson wrote:

> Hi,
> I would like to contribute the a fix to enable the Linux s390x platform.
> The fix was built against today?s mercurial master and pertains to the gcc
> atomic functions. I note the other architectures use inline assembler
> rather than the gcc builtin operations. Is this for historical reasons?

GCC builtin atomic operations are used when available, grep 
NGX_HAVE_GCC_ATOMIC for details.

-- 
Maxim Dounin
http://nginx.org/


From neale at sinenomine.net  Tue Oct 13 13:22:58 2015
From: neale at sinenomine.net (Neale Ferguson)
Date: Tue, 13 Oct 2015 13:22:58 +0000
Subject: Patch for s390x support
In-Reply-To: <20151013124408.GY30105@mdounin.ru>
References: <D24164B4.44E00%neale@sinenomine.net>
 <20151013124408.GY30105@mdounin.ru>
Message-ID: <D2427D0B.4508C%neale@sinenomine.net>

Thanks. I have the non-builtin version of the atomic operations written
for s390x, so on the off chance if someone wants to build using gcc < 4.1
then things will build/run. However, given then low probability of that is
it worth submitting the patch?

On 10/13/15, 8:44 AM, "nginx-devel on behalf of Maxim Dounin"
<nginx-devel-bounces at nginx.org on behalf of mdounin at mdounin.ru> wrote:

>Hello!
>
>On Mon, Oct 12, 2015 at 05:23:04PM ?, Neale Ferguson wrote:
>
>> Hi,
>> I would like to contribute the a fix to enable the Linux s390x platform.
>> The fix was built against today?s mercurial master and pertains to the
>>gcc
>> atomic functions. I note the other architectures use inline assembler
>> rather than the gcc builtin operations. Is this for historical reasons?
>
>GCC builtin atomic operations are used when available, grep
>NGX_HAVE_GCC_ATOMIC for details.


From bartw at xs4all.nl  Tue Oct 13 21:21:45 2015
From: bartw at xs4all.nl (Bart Warmerdam)
Date: Tue, 13 Oct 2015 23:21:45 +0200
Subject: Resource leak on error paths in thread pools
Message-ID: <1444771305.1669.2.camel@xs4all.nl>


It looks like in the error paths the attr variable is not destroyed.
Please consider adding this patch to the source base.

Regards,

B.


# HG changeset patch
# User bartw at xs4all.nl
# Date 1444770783 -7200
#      Tue Oct 13 23:13:03 2015 +0200
# Branch thread_unrelease_attr
# Node ID c2ae7364ec3f2251b8d734f3be7b62ea413dc36f
# Parent  2f34ea503ac4e015cc08f6efbb279b360eda609c
Release attr variable on exit

diff -r 2f34ea503ac4 -r c2ae7364ec3f src/core/ngx_thread_pool.c
--- a/src/core/ngx_thread_pool.c	Wed Oct 07 22:19:42 2015 +0300
+++ b/src/core/ngx_thread_pool.c	Tue Oct 13 23:13:03 2015 +0200
@@ -140,6 +140,7 @@
 #if 0
     err = pthread_attr_setstacksize(&attr, PTHREAD_STACK_MIN);
     if (err) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_ALERT, log, err,
                       "pthread_attr_setstacksize() failed");
         return NGX_ERROR;
@@ -149,6 +150,7 @@
     for (n = 0; n < tp->threads; n++) {
         err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
         if (err) {
+            (void) pthread_attr_destroy(&attr);
             ngx_log_error(NGX_LOG_ALERT, log, err,
                           "pthread_create() failed");
             return NGX_ERROR;
diff -r 2f34ea503ac4 -r c2ae7364ec3f src/os/unix/ngx_thread_mutex.c
--- a/src/os/unix/ngx_thread_mutex.c	Wed Oct 07 22:19:42 2015
+0300
+++ b/src/os/unix/ngx_thread_mutex.c	Tue Oct 13 23:13:03 2015
+0200
@@ -89,6 +89,7 @@
 
     err = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
     if (err != 0) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_EMERG, log, err,
                       "pthread_mutexattr_settype"
                       "(PTHREAD_MUTEX_ERRORCHECK) failed");
@@ -97,6 +98,7 @@
 
     err = pthread_mutex_init(mtx, &attr);
     if (err != 0) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_EMERG, log, err,
                       "pthread_mutex_init() failed");
         return NGX_ERROR;


From bartw at xs4all.nl  Tue Oct 13 21:58:04 2015
From: bartw at xs4all.nl (Bart Warmerdam)
Date: Tue, 13 Oct 2015 23:58:04 +0200
Subject: Error path does not close fd
Message-ID: <1444773484.19364.4.camel@xs4all.nl>



Hello,

In the ngx_daemon.c the dup2 result code is checked but the earlier
opened /dev/null handle is not closed in case of an error. Please
consider this path to add to the source base.

Regards,

B.


# HG changeset patch
# User Bart Warmerdam <bartw at xs4all.nl>
# Date 1444773372 -7200
#??????Tue Oct 13 23:56:12 2015 +0200
# Branch close_fd_on_error
# Node ID e1e25db76cdf7583f1145b91b9dbcdff417d6f16
# Parent??2f34ea503ac4e015cc08f6efbb279b360eda609c
Close file handle on error as well

diff -r 2f34ea503ac4 -r e1e25db76cdf src/os/unix/ngx_daemon.c
--- a/src/os/unix/ngx_daemon.c	Wed Oct 07 22:19:42 2015 +0300
+++ b/src/os/unix/ngx_daemon.c	Tue Oct 13 23:56:12 2015 +0200
@@ -9,6 +9,20 @@
?#include <ngx_core.h>
?
?
+static ngx_int_t
+ngx_close_handle(ngx_log_t *log, int fd)
+{
+????if (fd > STDERR_FILENO) {
+????????if (close(fd) == -1) {
+????????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "close()
failed");
+????????????return NGX_ERROR;
+????????}
+????}
+
+????return NGX_OK;
+}
+
+
?ngx_int_t
?ngx_daemon(ngx_log_t *log)
?{
@@ -44,26 +58,26 @@
?
?????if (dup2(fd, STDIN_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDIN)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?
?????if (dup2(fd, STDOUT_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDOUT)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?
?#if 0
?????if (dup2(fd, STDERR_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDERR)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?#endif
?
-????if (fd > STDERR_FILENO) {
-????????if (close(fd) == -1) {
-????????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "close()
failed");
-????????????return NGX_ERROR;
-????????}
+????if (ngx_close_handle(log, fd) == NGX_ERROR) {
+????????return NGX_ERROR;
?????}
?
?????return NGX_OK;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151013/26bee341/attachment.html>

From ru at nginx.com  Wed Oct 14 10:38:18 2015
From: ru at nginx.com (Ruslan Ermilov)
Date: Wed, 14 Oct 2015 13:38:18 +0300
Subject: Resource leak on error paths in thread pools
In-Reply-To: <1444771305.1669.2.camel@xs4all.nl>
References: <1444771305.1669.2.camel@xs4all.nl>
Message-ID: <20151014103818.GA75964@lo0.su>

On Tue, Oct 13, 2015 at 11:21:45PM +0200, Bart Warmerdam wrote:
> It looks like in the error paths the attr variable is not destroyed.
> Please consider adding this patch to the source base.

No thanks (please see below).

> # HG changeset patch
> # User bartw at xs4all.nl
> # Date 1444770783 -7200
> #      Tue Oct 13 23:13:03 2015 +0200
> # Branch thread_unrelease_attr
> # Node ID c2ae7364ec3f2251b8d734f3be7b62ea413dc36f
> # Parent  2f34ea503ac4e015cc08f6efbb279b360eda609c
> Release attr variable on exit
> 
> diff -r 2f34ea503ac4 -r c2ae7364ec3f src/core/ngx_thread_pool.c
> --- a/src/core/ngx_thread_pool.c	Wed Oct 07 22:19:42 2015 +0300
> +++ b/src/core/ngx_thread_pool.c	Tue Oct 13 23:13:03 2015 +0200
> @@ -140,6 +140,7 @@
>  #if 0
>      err = pthread_attr_setstacksize(&attr, PTHREAD_STACK_MIN);
>      if (err) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_ALERT, log, err,
>                        "pthread_attr_setstacksize() failed");
>          return NGX_ERROR;
> @@ -149,6 +150,7 @@
>      for (n = 0; n < tp->threads; n++) {
>          err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
>          if (err) {
> +            (void) pthread_attr_destroy(&attr);
>              ngx_log_error(NGX_LOG_ALERT, log, err,
>                            "pthread_create() failed");
>              return NGX_ERROR;

There's no leak here because if ngx_thread_pool_init() fails, the
worker process will exit with an error, effectively releasing all
resources.

We similarly don't care about destroying successfully created
threads if we fail creating the next thread.

> diff -r 2f34ea503ac4 -r c2ae7364ec3f src/os/unix/ngx_thread_mutex.c
> --- a/src/os/unix/ngx_thread_mutex.c	Wed Oct 07 22:19:42 2015
> +0300
> +++ b/src/os/unix/ngx_thread_mutex.c	Tue Oct 13 23:13:03 2015
> +0200
> @@ -89,6 +89,7 @@
>  
>      err = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
>      if (err != 0) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_EMERG, log, err,
>                        "pthread_mutexattr_settype"
>                        "(PTHREAD_MUTEX_ERRORCHECK) failed");
> @@ -97,6 +98,7 @@
>  
>      err = pthread_mutex_init(mtx, &attr);
>      if (err != 0) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_EMERG, log, err,
>                        "pthread_mutex_init() failed");
>          return NGX_ERROR;

Ditto, but you also misspelled pthread_mutexattr_destroy()
as pthread_attr_destroy().


From mhenrion at appnexus.com  Wed Oct 14 16:13:10 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Wed, 14 Oct 2015 16:13:10 +0000
Subject: Setting a custom header on a proxied request
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A5D1978D6@01-exch-nym2.ad.corp.appnexus.com>

Hey all,

I'm now contemplating implementing a full-blown upstream module instead of using the proxy module to achieve my goals here. Does that sound like a sensible solution?

Also, if I do go down that road, is it possible to have subrequests that use different upstreams? In order to send multiple requests for a single incoming request, I currently use a dummy location handler in the configuration, that invokes the "proxy_pass http://$arg_host" directive, and issue multiple subrequests to that location. Will I still need to do something similar with my own upstream module, or can I somehow create subrequests with different r->upstream pointers and have everything work as intended? From what I gathered so far, I don't quite see where the subrequest handling code would call ngx_http_upstream_init().

Thanks in advance,
Maxime

From mdounin at mdounin.ru  Wed Oct 14 16:26:06 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 14 Oct 2015 19:26:06 +0300
Subject: Patch for s390x support
In-Reply-To: <D2427D0B.4508C%neale@sinenomine.net>
References: <D24164B4.44E00%neale@sinenomine.net>
 <20151013124408.GY30105@mdounin.ru>
 <D2427D0B.4508C%neale@sinenomine.net>
Message-ID: <20151014162606.GG30105@mdounin.ru>

Hello!

On Tue, Oct 13, 2015 at 01:22:58PM +0000, Neale Ferguson wrote:

> Thanks. I have the non-builtin version of the atomic operations written
> for s390x, so on the off chance if someone wants to build using gcc < 4.1
> then things will build/run. However, given then low probability of that is
> it worth submitting the patch?

No.  The s390x platform certainly isn't that popular to justify an 
in-house atomic operations implementation.  In case someone wants 
to use a compiler without builtin atomic operations, the libatomic 
library can be used (or nginx can be used without atomic 
operations at all).

I would rather like to see C11 atomics support added - they aren't 
currently supported by nginx.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Wed Oct 14 16:32:51 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 14 Oct 2015 19:32:51 +0300
Subject: Setting a custom header on a proxied request
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <20151014163250.GH30105@mdounin.ru>

Hello!

On Mon, Oct 12, 2015 at 02:12:47PM +0000, Maxime Henrion wrote:

> Hi all,
> 
> 
> I'm currently writing an nginx module that sends a number of 
> subrequests to upstream servers based on application logic, and 
> then aggregates the results and send them back. That part 
> currently works just fine, thanks to the help I was given here 
> in the past.
> 
> However, I am now trying to add a custom header to the request 
> that is being proxied and currently failing at it. For various 
> technical reasons, I need to add a header that contains the IP 
> address and port (from the point of view of nginx) of the 
> upstream server currently being contacted. In order to get at 
> that address, I have hooked my module into the peer selection 
> mechanism, much like the keepalive, ip_hash modules and others 
> do. So far so good, I get at that address and can log it just 
> fine. However, I can't seem to be able to add it as a header to 
> my request.

[...]

>         h = ngx_list_push(&r->headers_in.headers);
>         if (h == NULL) {
>                 return NGX_ERROR;
>         }
>         h->hash = 1;
>         ngx_str_set(&h->key, "X-Shard-Addr");
>         h->value.data = ngx_pstrdup(r->pool, &peeraddr);
>         if (h->value.data == NULL) {
>                 return NGX_ERROR;
>         }
> 
>         h->value.len = peeraddr.len;
> 
>         return NGX_OK;
> }
> 
> So after getting at the actual address we're forwarding our 
> request to, I try to add a custom header to r->headers_in, on 
> the request object that I saved in my peer data.
> 
> I suspect I'm not actually dealing with the correct request 
> here, because no matter what I try, I can't get this code to 
> actually alter the headers of the forwarded request in any way.

What you are trying to do is wrong.  You shouldn't modify original 
request headers - they are expected to be from original request, 
and arbitrary modifications will cause problems with later 
processing.  Instead, consider using the proxy_set_header 
directive (http://nginx.org/r/proxy_set_header) to add headers to 
requests to upstream servers.

Additionally, this is not going to work as you are doing it too 
late.  You are trying to add headers from balancing function, but 
balancing happens when a request to upstream is already created.  
(And re-balancing due to proxy_next_upstream may select a 
different upstream server for the same request.)

-- 
Maxim Dounin
http://nginx.org/


From neale at sinenomine.net  Wed Oct 14 16:34:01 2015
From: neale at sinenomine.net (Neale Ferguson)
Date: Wed, 14 Oct 2015 16:34:01 +0000
Subject: Patch for s390x support
In-Reply-To: <20151014162606.GG30105@mdounin.ru>
References: <D24164B4.44E00%neale@sinenomine.net>
 <20151013124408.GY30105@mdounin.ru> <D2427D0B.4508C%neale@sinenomine.net>
 <20151014162606.GG30105@mdounin.ru>
Message-ID: <D243FC1A.4562B%neale@sinenomine.net>

Thanks. I?ll just let s390x default to the gcc builtins when available.

On 10/14/15, 12:26 PM, "nginx-devel on behalf of Maxim Dounin"
<nginx-devel-bounces at nginx.org on behalf of mdounin at mdounin.ru> wrote:

>
>No.  The s390x platform certainly isn't that popular to justify an
>in-house atomic operations implementation.  In case someone wants
>to use a compiler without builtin atomic operations, the libatomic
>library can be used (or nginx can be used without atomic
>operations at all).
>
>I would rather like to see C11 atomics support added - they aren't
>currently supported by nginx.


From mdounin at mdounin.ru  Wed Oct 14 16:34:45 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 14 Oct 2015 19:34:45 +0300
Subject: Setting a custom header on a proxied request
In-Reply-To: <561BDA02.9090908@nginx.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
 <561BDA02.9090908@nginx.com>
Message-ID: <20151014163444.GI30105@mdounin.ru>

Hello!

On Mon, Oct 12, 2015 at 05:04:18PM +0100, Andrew Hutchings wrote:

> Hi Maxime,
> 
> I could be wrong here (I'm not an expert at this section of code), but I
> believe what you need to set is:
> 
> r->upstream->headers_in
> 
> If this doesn't work please let me know and I'll build a test to try it
> out here.

No, r->upstream->headers_in is for headers got from an upstream 
server.

[...]

-- 
Maxim Dounin
http://nginx.org/


From mhenrion at appnexus.com  Wed Oct 14 16:56:35 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Wed, 14 Oct 2015 16:56:35 +0000
Subject: Setting a custom header on a proxied request
In-Reply-To: <20151014163250.GH30105@mdounin.ru>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>,
 <20151014163250.GH30105@mdounin.ru>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A5D1979AF@01-exch-nym2.ad.corp.appnexus.com>

Hello Maxim!

Thank you again for your answer.

I have read more about the upstream code as well as the the proxy module code, and now realize that headers are being serialized into a buffer a chain within the create_request callback, and that peer selection can happen after that.

I cannot however use proxy_set_header in this particular case, as I have no variable to use that would expand to the desired value (the actual IP address of the peer currently being contacted). To the best of my knowledge, the $upstream_addr variable is closest to what I need, except that it contains all the upstream addresses that were contacted during request processing.

Would you have any idea on how best to approach this problem? I was considering implementing a full-blown upstream module, but I have no evidence that the way the ngx_http_upstream.c code works is compatible with a scheme where headers would potentially need to be changed each time we select a new peer.

Thanks in advance,
Maxime

________________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Maxim Dounin [mdounin at mdounin.ru]
Sent: Wednesday, October 14, 2015 6:32 PM
To: nginx-devel at nginx.org
Subject: Re: Setting a custom header on a proxied request

Hello!

On Mon, Oct 12, 2015 at 02:12:47PM +0000, Maxime Henrion wrote:

> Hi all,
>
>
> I'm currently writing an nginx module that sends a number of
> subrequests to upstream servers based on application logic, and
> then aggregates the results and send them back. That part
> currently works just fine, thanks to the help I was given here
> in the past.
>
> However, I am now trying to add a custom header to the request
> that is being proxied and currently failing at it. For various
> technical reasons, I need to add a header that contains the IP
> address and port (from the point of view of nginx) of the
> upstream server currently being contacted. In order to get at
> that address, I have hooked my module into the peer selection
> mechanism, much like the keepalive, ip_hash modules and others
> do. So far so good, I get at that address and can log it just
> fine. However, I can't seem to be able to add it as a header to
> my request.

[...]

>         h = ngx_list_push(&r->headers_in.headers);
>         if (h == NULL) {
>                 return NGX_ERROR;
>         }
>         h->hash = 1;
>         ngx_str_set(&h->key, "X-Shard-Addr");
>         h->value.data = ngx_pstrdup(r->pool, &peeraddr);
>         if (h->value.data == NULL) {
>                 return NGX_ERROR;
>         }
>
>         h->value.len = peeraddr.len;
>
>         return NGX_OK;
> }
>
> So after getting at the actual address we're forwarding our
> request to, I try to add a custom header to r->headers_in, on
> the request object that I saved in my peer data.
>
> I suspect I'm not actually dealing with the correct request
> here, because no matter what I try, I can't get this code to
> actually alter the headers of the forwarded request in any way.

What you are trying to do is wrong.  You shouldn't modify original
request headers - they are expected to be from original request,
and arbitrary modifications will cause problems with later
processing.  Instead, consider using the proxy_set_header
directive (http://nginx.org/r/proxy_set_header) to add headers to
requests to upstream servers.

Additionally, this is not going to work as you are doing it too
late.  You are trying to add headers from balancing function, but
balancing happens when a request to upstream is already created.
(And re-balancing due to proxy_next_upstream may select a
different upstream server for the same request.)

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel


From mdounin at mdounin.ru  Wed Oct 14 17:09:27 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Wed, 14 Oct 2015 20:09:27 +0300
Subject: Setting a custom header on a proxied request
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A5D1979AF@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A5D1945DC@01-exch-nym2.ad.corp.appnexus.com>
 <20151014163250.GH30105@mdounin.ru>
 <EB010ACFBA76EF42ADCF5E994F19747A5D1979AF@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <20151014170927.GJ30105@mdounin.ru>

Hello!

On Wed, Oct 14, 2015 at 04:56:35PM +0000, Maxime Henrion wrote:

> Hello Maxim!
> 
> Thank you again for your answer.
> 
> I have read more about the upstream code as well as the the 
> proxy module code, and now realize that headers are being 
> serialized into a buffer a chain within the create_request 
> callback, and that peer selection can happen after that.
> 
> I cannot however use proxy_set_header in this particular case, 
> as I have no variable to use that would expand to the desired 
> value (the actual IP address of the peer currently being 
> contacted). To the best of my knowledge, the $upstream_addr 
> variable is closest to what I need, except that it contains all 
> the upstream addresses that were contacted during request 
> processing.
> 
> Would you have any idea on how best to approach this problem? I 
> was considering implementing a full-blown upstream module, but I 
> have no evidence that the way the ngx_http_upstream.c code works 
> is compatible with a scheme where headers would potentially need 
> to be changed each time we select a new peer.

What you are trying to do is not something easily doable with 
nginx upstream module.  It's designed to reuse the same request 
while connecting to multiple upstream servers from an upstream{} 
block, and it considers all upstream servers in a block to be 
identical - much like it happens when you are using a DNS name 
with multiple A records.

If you want requests to different servers be different, I would 
rather recommend using different upstream blocks and balancing 
requests between them using some external means - e.g., by using 
different locations or by using a proxy_pass with a variable.

-- 
Maxim Dounin
http://nginx.org/


From Runfeng.Wang at netbraintech.com  Thu Oct 15 01:11:14 2015
From: Runfeng.Wang at netbraintech.com (Runfeng.Wang at netbraintech.com)
Date: Thu, 15 Oct 2015 09:11:14 +0800
Subject: help
References: <mailman.5.1444824000.54956.nginx-devel@nginx.org>
Message-ID: <2015101509105380015867@netbraintech.com>

unsubscribe maillist for me plz



Runfeng.Wang at netbraintech.com
 
From: nginx-devel-request at nginx.org
Date: 2015-10-14 20:00
To: nginx-devel at nginx.org
Subject: nginx-devel Digest, Vol 72, Issue 10
Send nginx-devel mailing list submissions to
nginx-devel at nginx.org
 
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.nginx.org/mailman/listinfo/nginx-devel
or, via email, send a message with subject or body 'help' to
nginx-devel-request at nginx.org
 
You can reach the person managing the list at
nginx-devel-owner at nginx.org
 
When replying, please edit your Subject line so it is more specific
than "Re: Contents of nginx-devel digest..."
 
 
Today's Topics:
 
   1. Re: Patch for s390x support (Maxim Dounin)
   2. Re: Patch for s390x support (Neale Ferguson)
   3. Resource leak on error paths in thread pools (Bart Warmerdam)
   4. Error path does not close fd (Bart Warmerdam)
   5. Re: Resource leak on error paths in thread pools (Ruslan Ermilov)
 
 
----------------------------------------------------------------------
 
Message: 1
Date: Tue, 13 Oct 2015 15:44:08 +0300
From: Maxim Dounin <mdounin at mdounin.ru>
To: nginx-devel at nginx.org
Subject: Re: Patch for s390x support
Message-ID: <20151013124408.GY30105 at mdounin.ru>
Content-Type: text/plain; charset=utf-8
 
Hello!
 
On Mon, Oct 12, 2015 at 05:23:04PM +0000, Neale Ferguson wrote:
 
> Hi,
> I would like to contribute the a fix to enable the Linux s390x platform.
> The fix was built against today?s mercurial master and pertains to the gcc
> atomic functions. I note the other architectures use inline assembler
> rather than the gcc builtin operations. Is this for historical reasons?
 
GCC builtin atomic operations are used when available, grep 
NGX_HAVE_GCC_ATOMIC for details.
 
-- 
Maxim Dounin
http://nginx.org/
 
 
 
------------------------------
 
Message: 2
Date: Tue, 13 Oct 2015 13:22:58 +0000
From: Neale Ferguson <neale at sinenomine.net>
To: "nginx-devel at nginx.org" <nginx-devel at nginx.org>
Subject: Re: Patch for s390x support
Message-ID: <D2427D0B.4508C%neale at sinenomine.net>
Content-Type: text/plain; charset="utf-7"
 
Thanks. I have the non-builtin version of the atomic operations written
for s390x, so on the off chance if someone wants to build using gcc < 4.1
then things will build/run. However, given then low probability of that is
it worth submitting the patch?
 
On 10/13/15, 8:44 AM, "nginx-devel on behalf of Maxim Dounin"
<nginx-devel-bounces at nginx.org on behalf of mdounin at mdounin.ru> wrote:
 
>Hello!
>
>On Mon, Oct 12, 2015 at 05:23:04PM ?, Neale Ferguson wrote:
>
>> Hi,
>> I would like to contribute the a fix to enable the Linux s390x platform.
>> The fix was built against today?s mercurial master and pertains to the
>>gcc
>> atomic functions. I note the other architectures use inline assembler
>> rather than the gcc builtin operations. Is this for historical reasons?
>
>GCC builtin atomic operations are used when available, grep
>NGX_HAVE_GCC_ATOMIC for details.
 
 
 
------------------------------
 
Message: 3
Date: Tue, 13 Oct 2015 23:21:45 +0200
From: Bart Warmerdam <bartw at xs4all.nl>
To: nginx-devel at nginx.org
Subject: Resource leak on error paths in thread pools
Message-ID: <1444771305.1669.2.camel at xs4all.nl>
Content-Type: text/plain; charset="us-ascii"
 
 
It looks like in the error paths the attr variable is not destroyed.
Please consider adding this patch to the source base.
 
Regards,
 
B.
 
 
# HG changeset patch
# User bartw at xs4all.nl
# Date 1444770783 -7200
#      Tue Oct 13 23:13:03 2015 +0200
# Branch thread_unrelease_attr
# Node ID c2ae7364ec3f2251b8d734f3be7b62ea413dc36f
# Parent  2f34ea503ac4e015cc08f6efbb279b360eda609c
Release attr variable on exit
 
diff -r 2f34ea503ac4 -r c2ae7364ec3f src/core/ngx_thread_pool.c
--- a/src/core/ngx_thread_pool.c Wed Oct 07 22:19:42 2015 +0300
+++ b/src/core/ngx_thread_pool.c Tue Oct 13 23:13:03 2015 +0200
@@ -140,6 +140,7 @@
#if 0
     err = pthread_attr_setstacksize(&attr, PTHREAD_STACK_MIN);
     if (err) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_ALERT, log, err,
                       "pthread_attr_setstacksize() failed");
         return NGX_ERROR;
@@ -149,6 +150,7 @@
     for (n = 0; n < tp->threads; n++) {
         err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
         if (err) {
+            (void) pthread_attr_destroy(&attr);
             ngx_log_error(NGX_LOG_ALERT, log, err,
                           "pthread_create() failed");
             return NGX_ERROR;
diff -r 2f34ea503ac4 -r c2ae7364ec3f src/os/unix/ngx_thread_mutex.c
--- a/src/os/unix/ngx_thread_mutex.c Wed Oct 07 22:19:42 2015
+0300
+++ b/src/os/unix/ngx_thread_mutex.c Tue Oct 13 23:13:03 2015
+0200
@@ -89,6 +89,7 @@
     err = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
     if (err != 0) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_EMERG, log, err,
                       "pthread_mutexattr_settype"
                       "(PTHREAD_MUTEX_ERRORCHECK) failed");
@@ -97,6 +98,7 @@
     err = pthread_mutex_init(mtx, &attr);
     if (err != 0) {
+        (void) pthread_attr_destroy(&attr);
         ngx_log_error(NGX_LOG_EMERG, log, err,
                       "pthread_mutex_init() failed");
         return NGX_ERROR;
 
 
 
------------------------------
 
Message: 4
Date: Tue, 13 Oct 2015 23:58:04 +0200
From: Bart Warmerdam <bartw at xs4all.nl>
To: nginx-devel at nginx.org
Subject: Error path does not close fd
Message-ID: <1444773484.19364.4.camel at xs4all.nl>
Content-Type: text/plain; charset="iso-8859-1"
 
 
 
Hello,
 
In the ngx_daemon.c the dup2 result code is checked but the earlier
opened /dev/null handle is not closed in case of an error. Please
consider this path to add to the source base.
 
Regards,
 
B.
 
 
# HG changeset patch
# User Bart Warmerdam <bartw at xs4all.nl>
# Date 1444773372 -7200
#??????Tue Oct 13 23:56:12 2015 +0200
# Branch close_fd_on_error
# Node ID e1e25db76cdf7583f1145b91b9dbcdff417d6f16
# Parent??2f34ea503ac4e015cc08f6efbb279b360eda609c
Close file handle on error as well
 
diff -r 2f34ea503ac4 -r e1e25db76cdf src/os/unix/ngx_daemon.c
--- a/src/os/unix/ngx_daemon.c Wed Oct 07 22:19:42 2015 +0300
+++ b/src/os/unix/ngx_daemon.c Tue Oct 13 23:56:12 2015 +0200
@@ -9,6 +9,20 @@
?#include <ngx_core.h>
?
?
+static ngx_int_t
+ngx_close_handle(ngx_log_t *log, int fd)
+{
+????if (fd > STDERR_FILENO) {
+????????if (close(fd) == -1) {
+????????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "close()
failed");
+????????????return NGX_ERROR;
+????????}
+????}
+
+????return NGX_OK;
+}
+
+
?ngx_int_t
?ngx_daemon(ngx_log_t *log)
?{
@@ -44,26 +58,26 @@
?
?????if (dup2(fd, STDIN_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDIN)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?
?????if (dup2(fd, STDOUT_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDOUT)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?
?#if 0
?????if (dup2(fd, STDERR_FILENO) == -1) {
?????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "dup2(STDERR)
failed");
+????????ngx_close_handle(log, fd);
?????????return NGX_ERROR;
?????}
?#endif
?
-????if (fd > STDERR_FILENO) {
-????????if (close(fd) == -1) {
-????????????ngx_log_error(NGX_LOG_EMERG, log, ngx_errno, "close()
failed");
-????????????return NGX_ERROR;
-????????}
+????if (ngx_close_handle(log, fd) == NGX_ERROR) {
+????????return NGX_ERROR;
?????}
?
?????return NGX_OK;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151013/26bee341/attachment-0001.html>
 
------------------------------
 
Message: 5
Date: Wed, 14 Oct 2015 13:38:18 +0300
From: Ruslan Ermilov <ru at nginx.com>
To: nginx-devel at nginx.org
Subject: Re: Resource leak on error paths in thread pools
Message-ID: <20151014103818.GA75964 at lo0.su>
Content-Type: text/plain; charset=us-ascii
 
On Tue, Oct 13, 2015 at 11:21:45PM +0200, Bart Warmerdam wrote:
> It looks like in the error paths the attr variable is not destroyed.
> Please consider adding this patch to the source base.
 
No thanks (please see below).
 
> # HG changeset patch
> # User bartw at xs4all.nl
> # Date 1444770783 -7200
> #      Tue Oct 13 23:13:03 2015 +0200
> # Branch thread_unrelease_attr
> # Node ID c2ae7364ec3f2251b8d734f3be7b62ea413dc36f
> # Parent  2f34ea503ac4e015cc08f6efbb279b360eda609c
> Release attr variable on exit
> 
> diff -r 2f34ea503ac4 -r c2ae7364ec3f src/core/ngx_thread_pool.c
> --- a/src/core/ngx_thread_pool.c Wed Oct 07 22:19:42 2015 +0300
> +++ b/src/core/ngx_thread_pool.c Tue Oct 13 23:13:03 2015 +0200
> @@ -140,6 +140,7 @@
>  #if 0
>      err = pthread_attr_setstacksize(&attr, PTHREAD_STACK_MIN);
>      if (err) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_ALERT, log, err,
>                        "pthread_attr_setstacksize() failed");
>          return NGX_ERROR;
> @@ -149,6 +150,7 @@
>      for (n = 0; n < tp->threads; n++) {
>          err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
>          if (err) {
> +            (void) pthread_attr_destroy(&attr);
>              ngx_log_error(NGX_LOG_ALERT, log, err,
>                            "pthread_create() failed");
>              return NGX_ERROR;
 
There's no leak here because if ngx_thread_pool_init() fails, the
worker process will exit with an error, effectively releasing all
resources.
 
We similarly don't care about destroying successfully created
threads if we fail creating the next thread.
 
> diff -r 2f34ea503ac4 -r c2ae7364ec3f src/os/unix/ngx_thread_mutex.c
> --- a/src/os/unix/ngx_thread_mutex.c Wed Oct 07 22:19:42 2015
> +0300
> +++ b/src/os/unix/ngx_thread_mutex.c Tue Oct 13 23:13:03 2015
> +0200
> @@ -89,6 +89,7 @@
>  
>      err = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK);
>      if (err != 0) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_EMERG, log, err,
>                        "pthread_mutexattr_settype"
>                        "(PTHREAD_MUTEX_ERRORCHECK) failed");
> @@ -97,6 +98,7 @@
>  
>      err = pthread_mutex_init(mtx, &attr);
>      if (err != 0) {
> +        (void) pthread_attr_destroy(&attr);
>          ngx_log_error(NGX_LOG_EMERG, log, err,
>                        "pthread_mutex_init() failed");
>          return NGX_ERROR;
 
Ditto, but you also misspelled pthread_mutexattr_destroy()
as pthread_attr_destroy().
 
 
 
------------------------------
 
Subject: Digest Footer
 
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel
 
------------------------------
 
End of nginx-devel Digest, Vol 72, Issue 10
*******************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151015/abe3985e/attachment.html>

From steven.hartland at multiplay.co.uk  Fri Oct 16 00:09:49 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Fri, 16 Oct 2015 00:09:49 +0000
Subject: [PATCH] Support FreeBSD jails for testing
Message-ID: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>

# HG changeset patch
# User Steven Hartland <steven.hartland at multiplay.co.uk>
# Date 1444954080 0
#      Fri Oct 16 00:08:00 2015 +0000
# Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
# Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
Support FreeBSD jails for testing

Ensure the test directory is read and writable to the test user.

If you request 127.0.0.1 in a FreeBSD jail without specific access
to 127.0.0.1 then the socket binds to the interface address to
maintain compatibility. This results in the log entries being
from the bound interface address. To prevent failure compare
with the bound IP when requesting 127.0.0.1 in combined test.

diff -r 78b4e12e6efe -r c22d8299e704 access_log.t
--- a/access_log.t	Mon Oct 12 12:57:03 2015 +0300
+++ b/access_log.t	Fri Oct 16 00:08:00 2015 +0000
@@ -155,11 +155,17 @@ SKIP: {
 
 # verify that by default, 'combined' format is used, 'off' disables logging
 
-$log = $t->read_file('combined.log');
-like($log,
-	qr!^\Q127.0.0.1 - - [\E .*
-		\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
-	'default log format');
+SKIP: {
+	my $sock = IO::Socket::INET->new(LocalAddr => '127.0.0.1');
+	skip('127.0.0.1 local address required', 1) unless defined $sock;
+
+	$log = $t->read_file('combined.log');
+	my $addr = $sock->sockhost;
+	like($log,
+		qr!^\Q$addr - - [\E .*
+			\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
+		'default log format');
+}
 
 # verify that log filtering works
 
diff -r 78b4e12e6efe -r c22d8299e704 lib/Test/Nginx.pm
--- a/lib/Test/Nginx.pm	Mon Oct 12 12:57:03 2015 +0300
+++ b/lib/Test/Nginx.pm	Fri Oct 16 00:08:00 2015 +0000
@@ -43,6 +43,9 @@ sub new {
 		TMPDIR => 1
 	)
 		or die "Can't create temp directory: $!\n";
+
+	chmod(0777, $self->{_testdir})
+		or die "Can't chmod temp directory: $!\n";
 	$self->{_testdir} =~ s!\\!/!g if $^O eq 'MSWin32';
 	mkdir "$self->{_testdir}/logs"
 		or die "Can't create logs directory: $!\n";


From steven.hartland at multiplay.co.uk  Fri Oct 16 01:36:13 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Fri, 16 Oct 2015 02:36:13 +0100
Subject: Error handling from filter modules
Message-ID: <5620548D.8010601@multiplay.co.uk>

I'm making changes to a filter module and when it detected an error it
returned NGX_ERROR however the response generated to the client isn't
the expected 500 internal server error I would have expected given said
return.

So the question is do filters have to manually call
ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
expected that the upper layers should actually do the right thing and
ensure the client doesn't get a bad response generated from the current
state of r with no indication an error occurred?

	Regards
	Steve


From h2013111 at pilani.bits-pilani.ac.in  Fri Oct 16 06:02:42 2015
From: h2013111 at pilani.bits-pilani.ac.in (Ruchika Luthra)
Date: Fri, 16 Oct 2015 11:32:42 +0530
Subject: Debugging nginx quiet ugrade
Message-ID: <CACXoOBb6qqKDRze1fEyoq6PM-BugEz1VbqJeJBPpvZ4YU20KGg@mail.gmail.com>

Hi Everyone,

Can anyone please help me how to debug the failure the nginx quietpgrade?
Also I am unable to locate nginx.pid.oldbin. Could this be one of the
reason?

Thank you

-- 

Ruchika Luthra
Software Developer
Avni Networks
3rd Floor, #166, 19th Main, Sector 4, HSR Layout, Bengaluru, Karnataka
560102

luthraruchika at gmail.com
h2013111 at pilani.bits-pilani.ac.in
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151016/60ab16a7/attachment.html>

From nmav at redhat.com  Fri Oct 16 08:50:34 2015
From: nmav at redhat.com (Nikos Mavrogiannopoulos)
Date: Fri, 16 Oct 2015 10:50:34 +0200
Subject: OpenSSL PKCS#11 Engine cannot be reused in child process,
 worker SSL sessions fail
In-Reply-To: <CADkWibev3jm+cXaK_4e9e-UHbZToGxnVGJpu7Ur16+Pmuyez+A@mail.gmail.com>
References: <CADkWibev3jm+cXaK_4e9e-UHbZToGxnVGJpu7Ur16+Pmuyez+A@mail.gmail.com>
Message-ID: <1444985434.3415.10.camel@redhat.com>

On Sun, 2015-07-26 at 00:20 +0800, Anthony Alba wrote:
> Hi developers,
> 
> I am using nginx with an OpenSSL engine (Safenet Luna) which is a
> wrapper over PKCS#11.
> 
> The handles return by ENGINE_load_private_key cannot be used in child
> processes, aka, workers due to PKCS#11, thus causing SSL connection
> errors.
> The private key seems to be loaded in ngx_ssl_certificate(); is there
> a way to tell nginx to call this function per child process?

nginx should work transparently with the newest libp11 (0.3.0) and
engine_pkcs11 (0.2.0). Let me know, if not.

regards,
Nikos



From mdounin at mdounin.ru  Fri Oct 16 12:05:37 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 16 Oct 2015 15:05:37 +0300
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
Message-ID: <20151016120537.GE48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:

> # HG changeset patch
> # User Steven Hartland <steven.hartland at multiplay.co.uk>
> # Date 1444954080 0
> #      Fri Oct 16 00:08:00 2015 +0000
> # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
> # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
> Support FreeBSD jails for testing
> 
> Ensure the test directory is read and writable to the test user.
> 
> If you request 127.0.0.1 in a FreeBSD jail without specific access
> to 127.0.0.1 then the socket binds to the interface address to
> maintain compatibility. This results in the log entries being
> from the bound interface address. To prevent failure compare
> with the bound IP when requesting 127.0.0.1 in combined test.

This jails behaviour is known to cause many problems, in 
particular, it makes impossible nginx binary upgrades unless all 
listen sockets are explicitly bound to jail's IP address.

Fortunately, this was resolved several years ago by introduction 
of multi-IP jails.  You may try to use them for tests instead.

Adding quirks everywhere to support this brain-damaged "no 
127.0.0.1" case looks like a wrong way to go for me.  Especially 
given the fact that simple solution exists for years.

[...]

-- 
Maxim Dounin
http://nginx.org/


From arut at nginx.com  Fri Oct 16 12:16:23 2015
From: arut at nginx.com (Roman Arutyunyan)
Date: Fri, 16 Oct 2015 15:16:23 +0300
Subject: Error handling from filter modules
In-Reply-To: <5620548D.8010601@multiplay.co.uk>
References: <5620548D.8010601@multiplay.co.uk>
Message-ID: <81022093-EDDF-41D1-968D-3EA3C68DF9DC@nginx.com>

Hello Steve,

> On 16 Oct 2015, at 04:36, Steven Hartland <steven.hartland at multiplay.co.uk> wrote:
> 
> I'm making changes to a filter module

Header filter or body filter?

> and when it detected an error it
> returned NGX_ERROR however the response generated to the client isn't
> the expected 500 internal server error I would have expected given said
> return.

If you return NGX_ERROR from a filter handler, your request will be terminated
without sending any response to client.  If you want a certain error code
to be sent to client, you can either return it from your header filter handler
directly or call ngx_http_finalize_request().

> So the question is do filters have to manually call
> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
> expected that the upper layers should actually do the right thing and
> ensure the client doesn't get a bad response generated from the current
> state of r with no indication an error occurred?

Calling ngx_http_finalize_request() is the most correct way to return an
error from a filter handler.  It does some additional cleanup, which can
make sense in certain cases.

--
Roman Arutyunyan




From mdounin at mdounin.ru  Fri Oct 16 12:20:05 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 16 Oct 2015 15:20:05 +0300
Subject: Error handling from filter modules
In-Reply-To: <5620548D.8010601@multiplay.co.uk>
References: <5620548D.8010601@multiplay.co.uk>
Message-ID: <20151016122005.GG48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 02:36:13AM +0100, Steven Hartland wrote:

> I'm making changes to a filter module and when it detected an error it
> returned NGX_ERROR however the response generated to the client isn't
> the expected 500 internal server error I would have expected given said
> return.
> 
> So the question is do filters have to manually call
> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
> expected that the upper layers should actually do the right thing and
> ensure the client doesn't get a bad response generated from the current
> state of r with no indication an error occurred?

In filters, it's already to late to return anything in case of 
errors.  In body filters, it's way too late - the response was 
already partially sent.  And in header filters there is a chance that 
other filters allocated something response-specific, and an 
attempt to return a different response will break things.  So, 
when you return NGX_ERROR from a filter, the connection is just 
closed.

If you absolutely must return an error from a filter - like, e.g., 
not modified filter do for 416 responses - there is a special 
function to do this, ngx_http_filter_finalize_request().  It's 
very fragile though, and I wouldn't recommend using it unless you 
are sure you must do it.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Fri Oct 16 12:22:57 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 16 Oct 2015 15:22:57 +0300
Subject: Error handling from filter modules
In-Reply-To: <81022093-EDDF-41D1-968D-3EA3C68DF9DC@nginx.com>
References: <5620548D.8010601@multiplay.co.uk>
 <81022093-EDDF-41D1-968D-3EA3C68DF9DC@nginx.com>
Message-ID: <20151016122257.GH48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 03:16:23PM +0300, Roman Arutyunyan wrote:

> > So the question is do filters have to manually call
> > ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
> > expected that the upper layers should actually do the right thing and
> > ensure the client doesn't get a bad response generated from the current
> > state of r with no indication an error occurred?
> 
> Calling ngx_http_finalize_request() is the most correct way to return an
> error from a filter handler.  It does some additional cleanup, which can
> make sense in certain cases.

Huh?  The ngx_http_finalize_request() shouldn't be called from 
filters at all, this will cause incorrect reference counting and 
a segmentation fault as a most likely result.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Fri Oct 16 12:25:28 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 16 Oct 2015 15:25:28 +0300
Subject: Debugging nginx quiet ugrade
In-Reply-To: <CACXoOBb6qqKDRze1fEyoq6PM-BugEz1VbqJeJBPpvZ4YU20KGg@mail.gmail.com>
References: <CACXoOBb6qqKDRze1fEyoq6PM-BugEz1VbqJeJBPpvZ4YU20KGg@mail.gmail.com>
Message-ID: <20151016122528.GI48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 11:32:42AM +0530, Ruchika Luthra wrote:

> Hi Everyone,
> 
> Can anyone please help me how to debug the failure the nginx quietpgrade?
> Also I am unable to locate nginx.pid.oldbin. Could this be one of the
> reason?

You are using the wrong mailing list for the question.  This one is 
for development discussions.  Please use nginx at nginx.org mailing 
list instead, see http://nginx.org/en/support.html.

-- 
Maxim Dounin
http://nginx.org/


From steven.hartland at multiplay.co.uk  Fri Oct 16 16:49:39 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Fri, 16 Oct 2015 17:49:39 +0100
Subject: Error handling from filter modules
In-Reply-To: <5620548D.8010601@multiplay.co.uk>
References: <5620548D.8010601@multiplay.co.uk>
Message-ID: <56212AA3.6020304@multiplay.co.uk>

After doing some more digging is seems a better manual call would be:
return ngx_http_filter_finalize_request(r, &module, 
NGX_HTTP_INTERNAL_SERVER_ERROR);

But it still leaves the question is it ever sensible to allow bad 
responses to be returned to the client in the case NGX_ERROR is returned 
from a filter?

On 16/10/2015 02:36, Steven Hartland wrote:
> I'm making changes to a filter module and when it detected an error it
> returned NGX_ERROR however the response generated to the client isn't
> the expected 500 internal server error I would have expected given said
> return.
>
> So the question is do filters have to manually call
> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
> expected that the upper layers should actually do the right thing and
> ensure the client doesn't get a bad response generated from the current
> state of r with no indication an error occurred?
>
>     Regards
>     Steve
>


From steven.hartland at multiplay.co.uk  Fri Oct 16 17:15:30 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Fri, 16 Oct 2015 18:15:30 +0100
Subject: Error handling from filter modules
In-Reply-To: <20151016122005.GG48365@mdounin.ru>
References: <5620548D.8010601@multiplay.co.uk>
 <20151016122005.GG48365@mdounin.ru>
Message-ID: <562130B2.10001@multiplay.co.uk>

On 16/10/2015 13:20, Maxim Dounin wrote:
> Hello!
>
> On Fri, Oct 16, 2015 at 02:36:13AM +0100, Steven Hartland wrote:
>
>> I'm making changes to a filter module and when it detected an error it
>> returned NGX_ERROR however the response generated to the client isn't
>> the expected 500 internal server error I would have expected given said
>> return.
>>
>> So the question is do filters have to manually call
>> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
>> expected that the upper layers should actually do the right thing and
>> ensure the client doesn't get a bad response generated from the current
>> state of r with no indication an error occurred?
> In filters, it's already to late to return anything in case of
> errors.  In body filters, it's way too late - the response was
> already partially sent.  And in header filters there is a chance that
> other filters allocated something response-specific, and an
> attempt to return a different response will break things.  So,
> when you return NGX_ERROR from a filter, the connection is just
> closed.
There are a few core filters which seem to contradict this;

range does:
return ngx_http_range_not_satisfiable(r, ctx);

while image does:
return ngx_http_filter_finalize_request(r,
&ngx_http_image_filter_module,
NGX_HTTP_UNSUPPORTED_MEDIA_TYPE);

The range filter is the one I was looking at, picking back up the work 
you gave feedback on a while back with regards allow partial content 
responses to be used to satisfy range requests.

During testing for failures I triggered the NGX_ERROR case here:
https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_range_filter_module.c#L233

When I triggered this case client recieved an broken 200 response, where 
as if I did finalise the request I could deliver a 500 response, so was 
wondering if this would be correct in this situation?
> If you absolutely must return an error from a filter - like, e.g.,
> not modified filter do for 416 responses - there is a special
> function to do this, ngx_http_filter_finalize_request().  It's
> very fragile though, and I wouldn't recommend using it unless you
> are sure you must do it.
Yer I found that later but seems my mail's been playing up today, so 
sorry for the crossed replies.

Thanks for the feedback :)

     Regards
     Steve


From steven.hartland at multiplay.co.uk  Fri Oct 16 17:24:11 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Fri, 16 Oct 2015 18:24:11 +0100
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <20151016120537.GE48365@mdounin.ru>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru>
Message-ID: <562132BB.3030709@multiplay.co.uk>



On 16/10/2015 13:05, Maxim Dounin wrote:
> Hello!
>
> On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:
>
>> # HG changeset patch
>> # User Steven Hartland <steven.hartland at multiplay.co.uk>
>> # Date 1444954080 0
>> #      Fri Oct 16 00:08:00 2015 +0000
>> # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
>> # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
>> Support FreeBSD jails for testing
>>
>> Ensure the test directory is read and writable to the test user.
>>
>> If you request 127.0.0.1 in a FreeBSD jail without specific access
>> to 127.0.0.1 then the socket binds to the interface address to
>> maintain compatibility. This results in the log entries being
>> from the bound interface address. To prevent failure compare
>> with the bound IP when requesting 127.0.0.1 in combined test.
> This jails behaviour is known to cause many problems, in
> particular, it makes impossible nginx binary upgrades unless all
> listen sockets are explicitly bound to jail's IP address.
>
> Fortunately, this was resolved several years ago by introduction
> of multi-IP jails.  You may try to use them for tests instead.
>
> Adding quirks everywhere to support this brain-damaged "no
> 127.0.0.1" case looks like a wrong way to go for me.  Especially
> given the fact that simple solution exists for years.
>
> [...]
That doesn't fix the directory permission issue which causes pretty much 
every test to fail, so is this still an option for inclusion?

With regards to binding 127.0.0.1, yes its possible to bind it using 
multi IP, but doing so breaks security if you share it with the host, so 
its only possible in some situations and usually only for a proper loop 
back address which wouldn't be 127.0.0.1 just in that /24.

I do agree quirks aren't ideal, but as its only the one test I thought 
it would be nice to have given there's a simple and reliable way to 
correct said test.

With this in mind would you be up for making an exception in this case?

     Regards
     Steve


From arut at nginx.com  Fri Oct 16 18:16:31 2015
From: arut at nginx.com (Roman Arutyunyan)
Date: Fri, 16 Oct 2015 21:16:31 +0300
Subject: Error handling from filter modules
In-Reply-To: <20151016122257.GH48365@mdounin.ru>
References: <5620548D.8010601@multiplay.co.uk>
 <81022093-EDDF-41D1-968D-3EA3C68DF9DC@nginx.com>
 <20151016122257.GH48365@mdounin.ru>
Message-ID: <2655E99D-B292-4869-B13B-96972BF89379@nginx.com>


> On 16 Oct 2015, at 15:22, Maxim Dounin <mdounin at mdounin.ru> wrote:
> 
> Hello!
> 
> On Fri, Oct 16, 2015 at 03:16:23PM +0300, Roman Arutyunyan wrote:
> 
>>> So the question is do filters have to manually call
>>> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
>>> expected that the upper layers should actually do the right thing and
>>> ensure the client doesn't get a bad response generated from the current
>>> state of r with no indication an error occurred?
>> 
>> Calling ngx_http_finalize_request() is the most correct way to return an
>> error from a filter handler.  It does some additional cleanup, which can
>> make sense in certain cases.
> 
> Huh?  The ngx_http_finalize_request() shouldn't be called from 
> filters at all, this will cause incorrect reference counting and 
> a segmentation fault as a most likely result.

Sorry, of course, I meant ngx_http_filter_finalize_request().

--
Roman Arutyunyan




From kou at cozmixng.org  Sat Oct 17 12:44:32 2015
From: kou at cozmixng.org (Kouhei Sutou)
Date: Sat, 17 Oct 2015 21:44:32 +0900 (JST)
Subject: [PATCH] Win32: make buildable with MinGW-w64 gcc
Message-ID: <20151017.214432.472412871167194720.kou@cozmixng.org>

Hi,

The attached patch fixes the following warning:

    x86_64-w64-mingw32-gcc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/win32 -I objs \
            -o objs/src/core/ngx_connection.o \
            src/core/ngx_connection.c
    src/core/ngx_connection.c: In function 'ngx_close_idle_connections':
    src/core/ngx_connection.c:1178:21: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
             if (c[i].fd != -1 && c[i].idle) {
                         ^
    cc1: all warnings being treated as errors

Thanks,
--
kou
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nginx-win32-make-buildable-with-mingw-w64-gcc.patch
Type: text/x-patch
Size: 1434 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151017/7e729de4/attachment.bin>

From wantlamy at gmail.com  Sat Oct 17 15:43:29 2015
From: wantlamy at gmail.com (quink)
Date: Sat, 17 Oct 2015 23:43:29 +0800
Subject: [PATCH] ngx_event_pipe: remove unnecessary timer
Message-ID: <CA+mt-xojrJ7=jp74ptAV3DNXs6qb_8gqW9mw85JWfdfngGOO9A@mail.gmail.com>

A timer of 1 msec is not necessary take into account the resolutions of
ngx_time() and p->start.
If we don't consider the truncation error of ngx_time() and p->start, '1'
should be '1000'. If we
consider the truncation error, don't add timer if limit == 0 and discard '+
1'.

---
 src/event/ngx_event_pipe.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/event/ngx_event_pipe.c b/src/event/ngx_event_pipe.c
index 2d0e7d3..580a947 100644
--- a/src/event/ngx_event_pipe.c
+++ b/src/event/ngx_event_pipe.c
@@ -181,9 +181,9 @@ ngx_event_pipe_read_upstream(ngx_event_pipe_t *p)
                 limit = (off_t) p->limit_rate * (ngx_time() - p->start_sec
+ 1)
                         - p->read_length;

-                if (limit <= 0) {
+                if (limit < 0) {
                     p->upstream->read->delayed = 1;
-                    delay = (ngx_msec_t) (- limit * 1000 / p->limit_rate +
1);
+                    delay = (ngx_msec_t) (- limit * 1000 / p->limit_rate);
                     ngx_add_timer(p->upstream->read, delay);
                     break;
                 }
-- 
1.9.1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151017/7e82da87/attachment.html>

From vbart at nginx.com  Mon Oct 19 13:23:53 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 19 Oct 2015 13:23:53 +0000
Subject: [nginx] HTTP/2: fixed parsing of literal header fields without i...
Message-ID: <hg.0e37389c0bd5.1445261033.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/0e37389c0bd5
branches:  
changeset: 6260:0e37389c0bd5
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 19 16:16:47 2015 +0300
description:
HTTP/2: fixed parsing of literal header fields without indexing.

diffstat:

 src/http/v2/ngx_http_v2.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 2f34ea503ac4 -r 0e37389c0bd5 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Wed Oct 07 22:19:42 2015 +0300
+++ b/src/http/v2/ngx_http_v2.c	Mon Oct 19 16:16:47 2015 +0300
@@ -1233,7 +1233,7 @@ ngx_http_v2_state_header_block(ngx_http_
 
     } else {
         /* literal header field without indexing */
-        prefix = ngx_http_v2_prefix(3);
+        prefix = ngx_http_v2_prefix(4);
     }
 
     value = ngx_http_v2_parse_int(h2c, &pos, end, prefix);


From mdounin at mdounin.ru  Mon Oct 19 16:59:40 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 19:59:40 +0300
Subject: Error handling from filter modules
In-Reply-To: <562130B2.10001@multiplay.co.uk>
References: <5620548D.8010601@multiplay.co.uk>
 <20151016122005.GG48365@mdounin.ru>
 <562130B2.10001@multiplay.co.uk>
Message-ID: <20151019165940.GQ48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 06:15:30PM +0100, Steven Hartland wrote:

> On 16/10/2015 13:20, Maxim Dounin wrote:
> >Hello!
> >
> >On Fri, Oct 16, 2015 at 02:36:13AM +0100, Steven Hartland wrote:
> >
> >>I'm making changes to a filter module and when it detected an error it
> >>returned NGX_ERROR however the response generated to the client isn't
> >>the expected 500 internal server error I would have expected given said
> >>return.
> >>
> >>So the question is do filters have to manually call
> >>ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
> >>expected that the upper layers should actually do the right thing and
> >>ensure the client doesn't get a bad response generated from the current
> >>state of r with no indication an error occurred?
> >In filters, it's already to late to return anything in case of
> >errors.  In body filters, it's way too late - the response was
> >already partially sent.  And in header filters there is a chance that
> >other filters allocated something response-specific, and an
> >attempt to return a different response will break things.  So,
> >when you return NGX_ERROR from a filter, the connection is just
> >closed.
> There are a few core filters which seem to contradict this;
> 
> range does:
> return ngx_http_range_not_satisfiable(r, ctx);
> 
> while image does:
> return ngx_http_filter_finalize_request(r,
> &ngx_http_image_filter_module,
> NGX_HTTP_UNSUPPORTED_MEDIA_TYPE);

There is no contradiction here.  These are special cases which use 
ngx_http_filter_finalize_request(), as outlined later in the same 
message.

> The range filter is the one I was looking at, picking back up the work you
> gave feedback on a while back with regards allow partial content responses
> to be used to satisfy range requests.
> 
> During testing for failures I triggered the NGX_ERROR case here:
> https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_range_filter_module.c#L233
> 
> When I triggered this case client recieved an broken 200 response, where as
> if I did finalise the request I could deliver a 500 response, so was
> wondering if this would be correct in this situation?

When the NGX_ERROR case is triggered, no response is returned, the 
connection is just closed.  As this expected to happen if and only 
if we weren't able to allocate memory, this is believed to be 
perfectly correct behaviour.  Trying to do anything else is very 
unlikely to succeed, but likely to result in additional problems.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Mon Oct 19 17:38:34 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 20:38:34 +0300
Subject: [PATCH] ngx_event_pipe: remove unnecessary timer
In-Reply-To: <CA+mt-xojrJ7=jp74ptAV3DNXs6qb_8gqW9mw85JWfdfngGOO9A@mail.gmail.com>
References: <CA+mt-xojrJ7=jp74ptAV3DNXs6qb_8gqW9mw85JWfdfngGOO9A@mail.gmail.com>
Message-ID: <20151019173834.GT48365@mdounin.ru>

Hello!

On Sat, Oct 17, 2015 at 11:43:29PM +0800, quink wrote:

> A timer of 1 msec is not necessary take into account the resolutions of
> ngx_time() and p->start.
> If we don't consider the truncation error of ngx_time() and p->start, '1'
> should be '1000'. If we
> consider the truncation error, don't add timer if limit == 0 and discard '+
> 1'.
> 
> ---
>  src/event/ngx_event_pipe.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/src/event/ngx_event_pipe.c b/src/event/ngx_event_pipe.c
> index 2d0e7d3..580a947 100644
> --- a/src/event/ngx_event_pipe.c
> +++ b/src/event/ngx_event_pipe.c
> @@ -181,9 +181,9 @@ ngx_event_pipe_read_upstream(ngx_event_pipe_t *p)
>                  limit = (off_t) p->limit_rate * (ngx_time() - p->start_sec
> + 1)
>                          - p->read_length;
> 
> -                if (limit <= 0) {
> +                if (limit < 0) {
>                      p->upstream->read->delayed = 1;
> -                    delay = (ngx_msec_t) (- limit * 1000 / p->limit_rate +
> 1);
> +                    delay = (ngx_msec_t) (- limit * 1000 / p->limit_rate);
>                      ngx_add_timer(p->upstream->read, delay);
>                      break;
>                  }

If we've already exhausted allowed bandwidth (limit == 0), no 
further oprations will be possible.  So it doesn't make sense to 
try to read further data in such case.

Additionally, with the proposed patch things become just wrong, 
because as a result of the code limit is set to 0, which is later 
interpreted as "no limit is set" by the recv_chain() handler.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Mon Oct 19 17:57:27 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 20:57:27 +0300
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <562132BB.3030709@multiplay.co.uk>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru>
 <562132BB.3030709@multiplay.co.uk>
Message-ID: <20151019175727.GU48365@mdounin.ru>

Hello!

On Fri, Oct 16, 2015 at 06:24:11PM +0100, Steven Hartland wrote:

> On 16/10/2015 13:05, Maxim Dounin wrote:
> >Hello!
> >
> >On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:
> >
> >># HG changeset patch
> >># User Steven Hartland <steven.hartland at multiplay.co.uk>
> >># Date 1444954080 0
> >>#      Fri Oct 16 00:08:00 2015 +0000
> >># Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
> >># Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
> >>Support FreeBSD jails for testing
> >>
> >>Ensure the test directory is read and writable to the test user.
> >>
> >>If you request 127.0.0.1 in a FreeBSD jail without specific access
> >>to 127.0.0.1 then the socket binds to the interface address to
> >>maintain compatibility. This results in the log entries being
> >>from the bound interface address. To prevent failure compare
> >>with the bound IP when requesting 127.0.0.1 in combined test.
> >This jails behaviour is known to cause many problems, in
> >particular, it makes impossible nginx binary upgrades unless all
> >listen sockets are explicitly bound to jail's IP address.
> >
> >Fortunately, this was resolved several years ago by introduction
> >of multi-IP jails.  You may try to use them for tests instead.
> >
> >Adding quirks everywhere to support this brain-damaged "no
> >127.0.0.1" case looks like a wrong way to go for me.  Especially
> >given the fact that simple solution exists for years.
> >
> >[...]
> That doesn't fix the directory permission issue which causes pretty much
> every test to fail, so is this still an option for inclusion?

Directory premissions may vary depending on umask used.  If your 
umask breaks tests for you - you may try changing it while running 
tests.  It also shouldn't be important when running tests under 
non-privileged user.

I don't think this change should be added at all, and don't 
see how it's related to FreeBSD jails your patch says it's about.

> With regards to binding 127.0.0.1, yes its possible to bind it using multi
> IP, but doing so breaks security if you share it with the host, so its only
> possible in some situations and usually only for a proper loop back address
> which wouldn't be 127.0.0.1 just in that /24.

AFAIR, multi-IP jails used to provide per-jail loopback 
addresses.  But may be I'm wrong here and mistaken with wildcard 
addresses.

> I do agree quirks aren't ideal, but as its only the one test I thought it
> would be nice to have given there's a simple and reliable way to correct
> said test.
> 
> With this in mind would you be up for making an exception in this case?

As long as it's the only test affected we may consider it.
Sergey, could you please take a look?

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Mon Oct 19 18:27:45 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 18:27:45 +0000
Subject: [nginx] SSL: preserve default server context in connection (tick...
Message-ID: <hg.97f102a13f33.1445279265.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/97f102a13f33
branches:  
changeset: 6261:97f102a13f33
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Oct 19 21:22:38 2015 +0300
description:
SSL: preserve default server context in connection (ticket #235).

This context is needed for shared sessions cache to work in configurations
with multiple virtual servers sharing the same port.  Unfortunately, OpenSSL
does not provide an API to access the session context, thus storing it
separately.

In collaboration with Vladimir Homutov.

diffstat:

 src/event/ngx_event_openssl.c |  21 ++++++++-------------
 src/event/ngx_event_openssl.h |   1 +
 2 files changed, 9 insertions(+), 13 deletions(-)

diffs (84 lines):

diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c
--- a/src/event/ngx_event_openssl.c
+++ b/src/event/ngx_event_openssl.c
@@ -1038,6 +1038,8 @@ ngx_ssl_create_connection(ngx_ssl_t *ssl
     sc->buffer = ((flags & NGX_SSL_BUFFER) != 0);
     sc->buffer_size = ssl->buffer_size;
 
+    sc->session_ctx = ssl->ctx;
+
     sc->connection = SSL_new(ssl->ctx);
 
     if (sc->connection == NULL) {
@@ -2305,7 +2307,7 @@ ngx_ssl_new_session(ngx_ssl_conn_t *ssl_
 
     c = ngx_ssl_get_connection(ssl_conn);
 
-    ssl_ctx = SSL_get_SSL_CTX(ssl_conn);
+    ssl_ctx = c->ssl->session_ctx;
     shm_zone = SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_session_cache_index);
 
     cache = shm_zone->data;
@@ -2443,21 +2445,17 @@ ngx_ssl_get_cached_session(ngx_ssl_conn_
     ngx_ssl_sess_id_t        *sess_id;
     ngx_ssl_session_cache_t  *cache;
     u_char                    buf[NGX_SSL_MAX_SESSION_SIZE];
-#if (NGX_DEBUG)
     ngx_connection_t         *c;
-#endif
 
     hash = ngx_crc32_short(id, (size_t) len);
     *copy = 0;
 
-#if (NGX_DEBUG)
     c = ngx_ssl_get_connection(ssl_conn);
 
     ngx_log_debug2(NGX_LOG_DEBUG_EVENT, c->log, 0,
                    "ssl get session: %08XD:%d", hash, len);
-#endif
-
-    shm_zone = SSL_CTX_get_ex_data(SSL_get_SSL_CTX(ssl_conn),
+
+    shm_zone = SSL_CTX_get_ex_data(c->ssl->session_ctx,
                                    ngx_ssl_session_cache_index);
 
     cache = shm_zone->data;
@@ -2836,13 +2834,14 @@ ngx_ssl_session_ticket_key_callback(ngx_
     SSL_CTX                       *ssl_ctx;
     ngx_uint_t                     i;
     ngx_array_t                   *keys;
+    ngx_connection_t              *c;
     ngx_ssl_session_ticket_key_t  *key;
 #if (NGX_DEBUG)
     u_char                         buf[32];
-    ngx_connection_t              *c;
 #endif
 
-    ssl_ctx = SSL_get_SSL_CTX(ssl_conn);
+    c = ngx_ssl_get_connection(ssl_conn);
+    ssl_ctx = c->ssl->session_ctx;
 
     keys = SSL_CTX_get_ex_data(ssl_ctx, ngx_ssl_session_ticket_keys_index);
     if (keys == NULL) {
@@ -2851,10 +2850,6 @@ ngx_ssl_session_ticket_key_callback(ngx_
 
     key = keys->elts;
 
-#if (NGX_DEBUG)
-    c = ngx_ssl_get_connection(ssl_conn);
-#endif
-
     if (enc == 1) {
         /* encrypt session ticket */
 
diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h
--- a/src/event/ngx_event_openssl.h
+++ b/src/event/ngx_event_openssl.h
@@ -46,6 +46,7 @@ typedef struct {
 
 typedef struct {
     ngx_ssl_conn_t             *connection;
+    SSL_CTX                    *session_ctx;
 
     ngx_int_t                   last;
     ngx_buf_t                  *buf;


From mdounin at mdounin.ru  Mon Oct 19 19:27:38 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 19:27:38 +0000
Subject: [nginx] Fixed handling of empty root.
Message-ID: <hg.1063097b22b6.1445282858.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/1063097b22b6
branches:  
changeset: 6262:1063097b22b6
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Oct 19 21:28:02 2015 +0300
description:
Fixed handling of empty root.

Previously, a configuration with "root" set to an empty string tried test
a character at clcf->root.data[-1].  Additional test added to make sure
this won't happen.

Reported by Markus Linnala, see
http://mailman.nginx.org/pipermail/nginx-devel/2015-August/007210.html.

diffstat:

 src/http/ngx_http_core_module.c |  4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diffs (14 lines):

diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c
--- a/src/http/ngx_http_core_module.c
+++ b/src/http/ngx_http_core_module.c
@@ -4489,7 +4489,9 @@ ngx_http_core_root(ngx_conf_t *cf, ngx_c
     clcf->alias = alias ? clcf->name.len : 0;
     clcf->root = value[1];
 
-    if (!alias && clcf->root.data[clcf->root.len - 1] == '/') {
+    if (!alias && clcf->root.len > 0
+        && clcf->root.data[clcf->root.len - 1] == '/')
+    {
         clcf->root.len--;
     }
 


From mdounin at mdounin.ru  Mon Oct 19 19:27:41 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 19:27:41 +0000
Subject: [nginx] Fixed variables prefix comparison.
Message-ID: <hg.48c13a0824c5.1445282861.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/48c13a0824c5
branches:  
changeset: 6263:48c13a0824c5
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Oct 19 21:28:17 2015 +0300
description:
Fixed variables prefix comparison.

Variable names are not null-terminated, so using ngx_strncmp() without
extra length checks is wrong.

Reported by Markus Linnala,
http://mailman.nginx.org/pipermail/nginx-devel/2015-August/007211.html.

diffstat:

 src/http/ngx_http_variables.c |  38 ++++++++++++++++++++++++++------------
 1 files changed, 26 insertions(+), 12 deletions(-)

diffs (122 lines):

diff --git a/src/http/ngx_http_variables.c b/src/http/ngx_http_variables.c
--- a/src/http/ngx_http_variables.c
+++ b/src/http/ngx_http_variables.c
@@ -575,7 +575,7 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "http_", 5) == 0) {
+    if (name->len >= 5 && ngx_strncmp(name->data, "http_", 5) == 0) {
 
         if (ngx_http_variable_unknown_header_in(r, vv, (uintptr_t) name)
             == NGX_OK)
@@ -586,7 +586,7 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "sent_http_", 10) == 0) {
+    if (name->len >= 10 && ngx_strncmp(name->data, "sent_http_", 10) == 0) {
 
         if (ngx_http_variable_unknown_header_out(r, vv, (uintptr_t) name)
             == NGX_OK)
@@ -597,7 +597,7 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "upstream_http_", 14) == 0) {
+    if (name->len >= 14 && ngx_strncmp(name->data, "upstream_http_", 14) == 0) {
 
         if (ngx_http_upstream_header_variable(r, vv, (uintptr_t) name)
             == NGX_OK)
@@ -608,7 +608,7 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "cookie_", 7) == 0) {
+    if (name->len >= 7 && ngx_strncmp(name->data, "cookie_", 7) == 0) {
 
         if (ngx_http_variable_cookie(r, vv, (uintptr_t) name) == NGX_OK) {
             return vv;
@@ -617,7 +617,9 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "upstream_cookie_", 16) == 0) {
+    if (name->len >= 16
+        && ngx_strncmp(name->data, "upstream_cookie_", 16) == 0)
+    {
 
         if (ngx_http_upstream_cookie_variable(r, vv, (uintptr_t) name)
             == NGX_OK)
@@ -628,7 +630,7 @@ ngx_http_get_variable(ngx_http_request_t
         return NULL;
     }
 
-    if (ngx_strncmp(name->data, "arg_", 4) == 0) {
+    if (name->len >= 4 && ngx_strncmp(name->data, "arg_", 4) == 0) {
 
         if (ngx_http_variable_argument(r, vv, (uintptr_t) name) == NGX_OK) {
             return vv;
@@ -2535,21 +2537,27 @@ ngx_http_variables_init_vars(ngx_conf_t 
             }
         }
 
-        if (ngx_strncmp(v[i].name.data, "http_", 5) == 0) {
+        if (v[i].name.len >= 5
+            && ngx_strncmp(v[i].name.data, "http_", 5) == 0)
+        {
             v[i].get_handler = ngx_http_variable_unknown_header_in;
             v[i].data = (uintptr_t) &v[i].name;
 
             continue;
         }
 
-        if (ngx_strncmp(v[i].name.data, "sent_http_", 10) == 0) {
+        if (v[i].name.len >= 10
+            && ngx_strncmp(v[i].name.data, "sent_http_", 10) == 0)
+        {
             v[i].get_handler = ngx_http_variable_unknown_header_out;
             v[i].data = (uintptr_t) &v[i].name;
 
             continue;
         }
 
-        if (ngx_strncmp(v[i].name.data, "upstream_http_", 14) == 0) {
+        if (v[i].name.len >= 14
+            && ngx_strncmp(v[i].name.data, "upstream_http_", 14) == 0)
+        {
             v[i].get_handler = ngx_http_upstream_header_variable;
             v[i].data = (uintptr_t) &v[i].name;
             v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;
@@ -2557,14 +2565,18 @@ ngx_http_variables_init_vars(ngx_conf_t 
             continue;
         }
 
-        if (ngx_strncmp(v[i].name.data, "cookie_", 7) == 0) {
+        if (v[i].name.len >= 7
+            && ngx_strncmp(v[i].name.data, "cookie_", 7) == 0)
+        {
             v[i].get_handler = ngx_http_variable_cookie;
             v[i].data = (uintptr_t) &v[i].name;
 
             continue;
         }
 
-        if (ngx_strncmp(v[i].name.data, "upstream_cookie_", 16) == 0) {
+        if (v[i].name.len >= 16
+            && ngx_strncmp(v[i].name.data, "upstream_cookie_", 16) == 0)
+        {
             v[i].get_handler = ngx_http_upstream_cookie_variable;
             v[i].data = (uintptr_t) &v[i].name;
             v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;
@@ -2572,7 +2584,9 @@ ngx_http_variables_init_vars(ngx_conf_t 
             continue;
         }
 
-        if (ngx_strncmp(v[i].name.data, "arg_", 4) == 0) {
+        if (v[i].name.len >= 4
+            && ngx_strncmp(v[i].name.data, "arg_", 4) == 0)
+        {
             v[i].get_handler = ngx_http_variable_argument;
             v[i].data = (uintptr_t) &v[i].name;
             v[i].flags = NGX_HTTP_VAR_NOCACHEABLE;


From mdounin at mdounin.ru  Mon Oct 19 20:05:20 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 20:05:20 +0000
Subject: [nginx] Style: unneeded casts of cf->args->elts removed.
Message-ID: <hg.7ac57369036c.1445285120.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/7ac57369036c
branches:  
changeset: 6264:7ac57369036c
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Mon Oct 19 23:04:49 2015 +0300
description:
Style: unneeded casts of cf->args->elts removed.

diffstat:

 src/core/nginx.c |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (21 lines):

diff --git a/src/core/nginx.c b/src/core/nginx.c
--- a/src/core/nginx.c
+++ b/src/core/nginx.c
@@ -1146,7 +1146,7 @@ ngx_set_user(ngx_conf_t *cf, ngx_command
         return NGX_CONF_OK;
     }
 
-    value = (ngx_str_t *) cf->args->elts;
+    value = cf->args->elts;
 
     ccf->username = (char *) value[1].data;
 
@@ -1352,7 +1352,7 @@ ngx_set_worker_processes(ngx_conf_t *cf,
         return "is duplicate";
     }
 
-    value = (ngx_str_t *) cf->args->elts;
+    value = cf->args->elts;
 
     if (ngx_strcmp(value[1].data, "auto") == 0) {
         ccf->worker_processes = ngx_ncpu;


From mdounin at mdounin.ru  Mon Oct 19 20:52:46 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 20:52:46 +0000
Subject: [nginx] Win32: fixed build with MinGW and MinGW-w64 gcc.
Message-ID: <hg.954b67727af3.1445287966.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/954b67727af3
branches:  
changeset: 6265:954b67727af3
user:      Kouhei Sutou <kou at cozmixng.org>
date:      Sat Oct 17 21:41:02 2015 +0900
description:
Win32: fixed build with MinGW and MinGW-w64 gcc.

This change fixes the "comparison between signed and unsigned integer
expressions" warning, introduced in 5e6142609e48 (1.9.4).

diffstat:

 src/core/ngx_connection.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/src/core/ngx_connection.c b/src/core/ngx_connection.c
--- a/src/core/ngx_connection.c
+++ b/src/core/ngx_connection.c
@@ -1175,7 +1175,7 @@ ngx_close_idle_connections(ngx_cycle_t *
 
         /* THREAD: lock */
 
-        if (c[i].fd != -1 && c[i].idle) {
+        if (c[i].fd != (ngx_socket_t) -1 && c[i].idle) {
             c[i].close = 1;
             c[i].read->handler(c[i].read);
         }


From mdounin at mdounin.ru  Mon Oct 19 20:54:35 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 19 Oct 2015 23:54:35 +0300
Subject: [PATCH] Win32: make buildable with MinGW-w64 gcc
In-Reply-To: <20151017.214432.472412871167194720.kou@cozmixng.org>
References: <20151017.214432.472412871167194720.kou@cozmixng.org>
Message-ID: <20151019205435.GV48365@mdounin.ru>

Hello!

On Sat, Oct 17, 2015 at 09:44:32PM +0900, Kouhei Sutou wrote:

> Hi,
> 
> The attached patch fixes the following warning:
> 
>     x86_64-w64-mingw32-gcc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/win32 -I objs \
>             -o objs/src/core/ngx_connection.o \
>             src/core/ngx_connection.c
>     src/core/ngx_connection.c: In function 'ngx_close_idle_connections':
>     src/core/ngx_connection.c:1178:21: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
>              if (c[i].fd != -1 && c[i].idle) {
>                          ^
>     cc1: all warnings being treated as errors

[...]

Thanks, and this manifests itself with MinGW gcc as well.  
Committed with some changes to commit log:

http://hg.nginx.org/nginx/rev/954b67727af3

-- 
Maxim Dounin
http://nginx.org/


From pluknet at nginx.com  Mon Oct 19 21:26:41 2015
From: pluknet at nginx.com (Sergey Kandaurov)
Date: Tue, 20 Oct 2015 00:26:41 +0300
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <20151019175727.GU48365@mdounin.ru>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru> <562132BB.3030709@multiplay.co.uk>
 <20151019175727.GU48365@mdounin.ru>
Message-ID: <34974603-7F23-496F-94B0-F4715EB548D2@nginx.com>

On Oct 19, 2015, at 8:57 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
> Hello!
> 
> On Fri, Oct 16, 2015 at 06:24:11PM +0100, Steven Hartland wrote:
> 
>> On 16/10/2015 13:05, Maxim Dounin wrote:
>>> Hello!
>>> 
>>> On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:
>>> 
>>>> # HG changeset patch
>>>> # User Steven Hartland <steven.hartland at multiplay.co.uk>
>>>> # Date 1444954080 0
>>>> #      Fri Oct 16 00:08:00 2015 +0000
>>>> # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
>>>> # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
>>>> Support FreeBSD jails for testing
>>>> 
>>>> Ensure the test directory is read and writable to the test user.
>>>> 
>>>> If you request 127.0.0.1 in a FreeBSD jail without specific access
>>>> to 127.0.0.1 then the socket binds to the interface address to
>>>> maintain compatibility. This results in the log entries being
>>>> from the bound interface address. To prevent failure compare
>>>> with the bound IP when requesting 127.0.0.1 in combined test.
>>> This jails behaviour is known to cause many problems, in
>>> particular, it makes impossible nginx binary upgrades unless all
>>> listen sockets are explicitly bound to jail's IP address.
>>> 
>>> Fortunately, this was resolved several years ago by introduction
>>> of multi-IP jails.  You may try to use them for tests instead.
>>> 
>>> Adding quirks everywhere to support this brain-damaged "no
>>> 127.0.0.1" case looks like a wrong way to go for me.  Especially
>>> given the fact that simple solution exists for years.
>>> 
>>> [...]
>> That doesn't fix the directory permission issue which causes pretty much
>> every test to fail, so is this still an option for inclusion?
> 
> Directory premissions may vary depending on umask used.  If your 
> umask breaks tests for you - you may try changing it while running 
> tests.  It also shouldn't be important when running tests under 
> non-privileged user.
> 
> I don't think this change should be added at all, and don't 
> see how it?s related to FreeBSD jails your patch says it's about.

Same here.

> 
>> With regards to binding 127.0.0.1, yes its possible to bind it using multi
>> IP, but doing so breaks security if you share it with the host, so its only
>> possible in some situations and usually only for a proper loop back address
>> which wouldn't be 127.0.0.1 just in that /24.
> 
> AFAIR, multi-IP jails used to provide per-jail loopback 
> addresses.  But may be I'm wrong here and mistaken with wildcard 
> addresses.
> 
>> I do agree quirks aren't ideal, but as its only the one test I thought it
>> would be nice to have given there's a simple and reliable way to correct
>> said test.
>> 
>> With this in mind would you be up for making an exception in this case?
> 
> As long as it's the only test affected we may consider it.
> Sergey, could you please take a look?

I have no strong objections, but see below.

: # HG changeset patch
: # User Steven Hartland <steven.hartland at multiplay.co.uk>
: # Date 1444954080 0
: #      Fri Oct 16 00:08:00 2015 +0000
: # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
: # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
: Support FreeBSD jails for testing

Style: missing prefix ?Tests: ? used for tests, and a trailing dot.

: 
: Ensure the test directory is read and writable to the test user.

Not applicable as discussed above.

: 
: If you request 127.0.0.1 in a FreeBSD jail without specific access
: to 127.0.0.1 then the socket binds to the interface address to
: maintain compatibility. This results in the log entries being
: from the bound interface address. To prevent failure compare
: with the bound IP when requesting 127.0.0.1 in combined test.

I?d rather refer to ea574aae77cb for brevity.

: 
: diff -r 78b4e12e6efe -r c22d8299e704 access_log.t
: --- a/access_log.t	Mon Oct 12 12:57:03 2015 +0300
: +++ b/access_log.t	Fri Oct 16 00:08:00 2015 +0000
: @@ -155,11 +155,17 @@ SKIP: {
: 
: # verify that by default, 'combined' format is used, 'off' disables logging
: 
: -$log = $t->read_file('combined.log');
: -like($log,
: -	qr!^\Q127.0.0.1 - - [\E .*
: -		\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
: -	'default log format');
: +SKIP: {
: +	my $sock = IO::Socket::INET->new(LocalAddr => '127.0.0.1');
: +	skip(?127.0.0.1 local address required', 1) unless defined $sock;

This looks like an excessive step.
If perl could not successfully bind to 127.0.0.1 (or whatever it thinks to),
nginx won?t start as well with failure effectively skipping all the tests.

: +
: +	$log = $t->read_file('combined.log');
: +	my $addr = $sock->sockhost;

I suggest to use a method with explicit parentheses,
also for consistency with the rest nginx test suite.

: +	like($log,
: +		qr!^\Q$addr - - [\E .*
: +			\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
: +		'default log format');
: +}
: 
: # verify that log filtering works
: 

I intend to commit this change:

# HG changeset patch
# User Sergey Kandaurov <pluknet at nginx.com>
# Date 1445289338 -10800
#      Tue Oct 20 00:15:38 2015 +0300
# Node ID 0af5555e0b7b130d7162d304b074932331267882
# Parent  cdd3659d114487d55119e1bb45f9a5fb44dbf79c
Tests: unbreak access_log tests in jails.

See ea574aae77cb for details.

Reported by Steven Hartland, see
http://mailman.nginx.org/pipermail/nginx-devel/2015-October/007393.html.

diff -r cdd3659d1144 -r 0af5555e0b7b access_log.t
--- a/access_log.t	Mon Oct 19 20:43:22 2015 +0300
+++ b/access_log.t	Tue Oct 20 00:15:38 2015 +0300
@@ -155,9 +155,11 @@ SKIP: {
 
 # verify that by default, 'combined' format is used, 'off' disables logging
 
+my $addr = IO::Socket::INET->new(LocalAddr => '127.0.0.1')->sockhost();
+
 $log = $t->read_file('combined.log');
 like($log,
-	qr!^\Q127.0.0.1 - - [\E .*
+	qr!^\Q$addr - - [\E .*
 		\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
 	'default log format');
 

-- 
Sergey Kandaurov


From steven.hartland at multiplay.co.uk  Mon Oct 19 22:02:53 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Mon, 19 Oct 2015 23:02:53 +0100
Subject: Error handling from filter modules
In-Reply-To: <20151019165940.GQ48365@mdounin.ru>
References: <5620548D.8010601@multiplay.co.uk>
 <20151016122005.GG48365@mdounin.ru> <562130B2.10001@multiplay.co.uk>
 <20151019165940.GQ48365@mdounin.ru>
Message-ID: <5625688D.9030004@multiplay.co.uk>



On 19/10/2015 17:59, Maxim Dounin wrote:
> Hello!
>
> On Fri, Oct 16, 2015 at 06:15:30PM +0100, Steven Hartland wrote:
>
>> On 16/10/2015 13:20, Maxim Dounin wrote:
>>> Hello!
>>>
>>> On Fri, Oct 16, 2015 at 02:36:13AM +0100, Steven Hartland wrote:
>>>
>>>> I'm making changes to a filter module and when it detected an error it
>>>> returned NGX_ERROR however the response generated to the client isn't
>>>> the expected 500 internal server error I would have expected given said
>>>> return.
>>>>
>>>> So the question is do filters have to manually call
>>>> ngx_http_finalize_request(r, NGX_HTTP_INTERNAL_SERVER_ERROR); or is it
>>>> expected that the upper layers should actually do the right thing and
>>>> ensure the client doesn't get a bad response generated from the current
>>>> state of r with no indication an error occurred?
>>> In filters, it's already to late to return anything in case of
>>> errors.  In body filters, it's way too late - the response was
>>> already partially sent.  And in header filters there is a chance that
>>> other filters allocated something response-specific, and an
>>> attempt to return a different response will break things.  So,
>>> when you return NGX_ERROR from a filter, the connection is just
>>> closed.
>> There are a few core filters which seem to contradict this;
>>
>> range does:
>> return ngx_http_range_not_satisfiable(r, ctx);
>>
>> while image does:
>> return ngx_http_filter_finalize_request(r,
>> &ngx_http_image_filter_module,
>> NGX_HTTP_UNSUPPORTED_MEDIA_TYPE);
> There is no contradiction here.  These are special cases which use
> ngx_http_filter_finalize_request(), as outlined later in the same
> message.
>
>> The range filter is the one I was looking at, picking back up the work you
>> gave feedback on a while back with regards allow partial content responses
>> to be used to satisfy range requests.
>>
>> During testing for failures I triggered the NGX_ERROR case here:
>> https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_range_filter_module.c#L233
>>
>> When I triggered this case client recieved an broken 200 response, where as
>> if I did finalise the request I could deliver a 500 response, so was
>> wondering if this would be correct in this situation?
> When the NGX_ERROR case is triggered, no response is returned, the
> connection is just closed.
It does seem like the connection to the client is just closed however a 
206 response is logged for the request in the access log, hence my 
confusion; is that expected?
> As this expected to happen if and only
> if we weren't able to allocate memory, this is believed to be
> perfectly correct behaviour.  Trying to do anything else is very
> unlikely to succeed, but likely to result in additional problems.
>
In the current code yes, but in the patch that I was testing there's one 
additional case which can return an error. This was caused by an 
intentional configuration error (test designed to confirm the behaviour 
in this case).

Its likely going to be difficult to discuss fully without the patch to 
comment against so I'll finalise that and get it posted as soon as I can.

Thanks for all the feedback so far :)

     Regards
     Steve


From steven.hartland at multiplay.co.uk  Mon Oct 19 22:39:27 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Mon, 19 Oct 2015 23:39:27 +0100
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <20151019175727.GU48365@mdounin.ru>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru> <562132BB.3030709@multiplay.co.uk>
 <20151019175727.GU48365@mdounin.ru>
Message-ID: <5625711F.208@multiplay.co.uk>


On 19/10/2015 18:57, Maxim Dounin wrote:
> Hello!
>
> On Fri, Oct 16, 2015 at 06:24:11PM +0100, Steven Hartland wrote:
>
>> On 16/10/2015 13:05, Maxim Dounin wrote:
>>> Hello!
>>>
>>> On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:
>>>
>>>> # HG changeset patch
>>>> # User Steven Hartland <steven.hartland at multiplay.co.uk>
>>>> # Date 1444954080 0
>>>> #      Fri Oct 16 00:08:00 2015 +0000
>>>> # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
>>>> # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
>>>> Support FreeBSD jails for testing
>>>>
>>>> Ensure the test directory is read and writable to the test user.
>>>>
>>>> If you request 127.0.0.1 in a FreeBSD jail without specific access
>>>> to 127.0.0.1 then the socket binds to the interface address to
>>>> maintain compatibility. This results in the log entries being
>>> >from the bound interface address. To prevent failure compare
>>>> with the bound IP when requesting 127.0.0.1 in combined test.
>>> This jails behaviour is known to cause many problems, in
>>> particular, it makes impossible nginx binary upgrades unless all
>>> listen sockets are explicitly bound to jail's IP address.
>>>
>>> Fortunately, this was resolved several years ago by introduction
>>> of multi-IP jails.  You may try to use them for tests instead.
>>>
>>> Adding quirks everywhere to support this brain-damaged "no
>>> 127.0.0.1" case looks like a wrong way to go for me.  Especially
>>> given the fact that simple solution exists for years.
>>>
>>> [...]
>> That doesn't fix the directory permission issue which causes pretty much
>> every test to fail, so is this still an option for inclusion?
> Directory premissions may vary depending on umask used.  If your
> umask breaks tests for you - you may try changing it while running
> tests.  It also shouldn't be important when running tests under
> non-privileged user.
Confirmed it doesn't effect a non-privileged user.

The default umask is 022 hence the issue if its run as privileged user 
with the default user as nobody which will have no access to the created 
directory.
> I don't think this change should be added at all, and don't
> see how it's related to FreeBSD jails your patch says it's about.
Fair comment with regards relevance, it clearly would prevent the tests 
being run as a privileged user on any Unix platform which has a 022 
umask not just a FreeBSD jail.

Something to mention in the README, if you don't like the chmod?
>
>> With regards to binding 127.0.0.1, yes its possible to bind it using multi
>> IP, but doing so breaks security if you share it with the host, so its only
>> possible in some situations and usually only for a proper loop back address
>> which wouldn't be 127.0.0.1 just in that /24.
> AFAIR, multi-IP jails used to provide per-jail loopback
> addresses.  But may be I'm wrong here and mistaken with wildcard
> addresses.
Not aware of that, you can assign any address you like but it still 
needs to consciously done and any sharing with the host has the 
potential to cause issues.

Maybe you're thinking of vnet support which provides a jail with their 
own virtual network stack, however that requires a kernel compiled with 
the VIMAGE option.
>
>> I do agree quirks aren't ideal, but as its only the one test I thought it
>> would be nice to have given there's a simple and reliable way to correct
>> said test.
>>
>> With this in mind would you be up for making an exception in this case?
> As long as it's the only test affected we may consider it.
> Sergey, could you please take a look?
>


From steven.hartland at multiplay.co.uk  Mon Oct 19 22:44:59 2015
From: steven.hartland at multiplay.co.uk (Steven Hartland)
Date: Mon, 19 Oct 2015 23:44:59 +0100
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <34974603-7F23-496F-94B0-F4715EB548D2@nginx.com>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru> <562132BB.3030709@multiplay.co.uk>
 <20151019175727.GU48365@mdounin.ru>
 <34974603-7F23-496F-94B0-F4715EB548D2@nginx.com>
Message-ID: <5625726B.1050406@multiplay.co.uk>

On 19/10/2015 22:26, Sergey Kandaurov wrote:
> On Oct 19, 2015, at 8:57 PM, Maxim Dounin <mdounin at mdounin.ru> wrote:
>> Hello!
>>
>> On Fri, Oct 16, 2015 at 06:24:11PM +0100, Steven Hartland wrote:
>>
>>> On 16/10/2015 13:05, Maxim Dounin wrote:
>>>> Hello!
>>>>
>>>> On Fri, Oct 16, 2015 at 12:09:49AM +0000, Steven Hartland wrote:
>>>>
>>>>> # HG changeset patch
>>>>> # User Steven Hartland <steven.hartland at multiplay.co.uk>
>>>>> # Date 1444954080 0
>>>>> #      Fri Oct 16 00:08:00 2015 +0000
>>>>> # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
>>>>> # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
>>>>> Support FreeBSD jails for testing
>>>>>
>>>>> Ensure the test directory is read and writable to the test user.
>>>>>
>>>>> If you request 127.0.0.1 in a FreeBSD jail without specific access
>>>>> to 127.0.0.1 then the socket binds to the interface address to
>>>>> maintain compatibility. This results in the log entries being
>>>>> from the bound interface address. To prevent failure compare
>>>>> with the bound IP when requesting 127.0.0.1 in combined test.
>>>> This jails behaviour is known to cause many problems, in
>>>> particular, it makes impossible nginx binary upgrades unless all
>>>> listen sockets are explicitly bound to jail's IP address.
>>>>
>>>> Fortunately, this was resolved several years ago by introduction
>>>> of multi-IP jails.  You may try to use them for tests instead.
>>>>
>>>> Adding quirks everywhere to support this brain-damaged "no
>>>> 127.0.0.1" case looks like a wrong way to go for me.  Especially
>>>> given the fact that simple solution exists for years.
>>>>
>>>> [...]
>>> That doesn't fix the directory permission issue which causes pretty much
>>> every test to fail, so is this still an option for inclusion?
>> Directory premissions may vary depending on umask used.  If your
>> umask breaks tests for you - you may try changing it while running
>> tests.  It also shouldn't be important when running tests under
>> non-privileged user.
>>
>> I don't think this change should be added at all, and don't
>> see how it?s related to FreeBSD jails your patch says it's about.
> Same here.
>
>>> With regards to binding 127.0.0.1, yes its possible to bind it using multi
>>> IP, but doing so breaks security if you share it with the host, so its only
>>> possible in some situations and usually only for a proper loop back address
>>> which wouldn't be 127.0.0.1 just in that /24.
>> AFAIR, multi-IP jails used to provide per-jail loopback
>> addresses.  But may be I'm wrong here and mistaken with wildcard
>> addresses.
>>
>>> I do agree quirks aren't ideal, but as its only the one test I thought it
>>> would be nice to have given there's a simple and reliable way to correct
>>> said test.
>>>
>>> With this in mind would you be up for making an exception in this case?
>> As long as it's the only test affected we may consider it.
>> Sergey, could you please take a look?
> I have no strong objections, but see below.
>
> : # HG changeset patch
> : # User Steven Hartland <steven.hartland at multiplay.co.uk>
> : # Date 1444954080 0
> : #      Fri Oct 16 00:08:00 2015 +0000
> : # Node ID c22d8299e7040e0de6f85b4e96d0dd953f7af644
> : # Parent  78b4e12e6efe642aff591234db0f0b040cae9b5e
> : Support FreeBSD jails for testing
>
> Style: missing prefix ?Tests: ? used for tests, and a trailing dot.
>
> :
> : Ensure the test directory is read and writable to the test user.
>
> Not applicable as discussed above.
>
> :
> : If you request 127.0.0.1 in a FreeBSD jail without specific access
> : to 127.0.0.1 then the socket binds to the interface address to
> : maintain compatibility. This results in the log entries being
> : from the bound interface address. To prevent failure compare
> : with the bound IP when requesting 127.0.0.1 in combined test.
>
> I?d rather refer to ea574aae77cb for brevity.
>
> :
> : diff -r 78b4e12e6efe -r c22d8299e704 access_log.t
> : --- a/access_log.t	Mon Oct 12 12:57:03 2015 +0300
> : +++ b/access_log.t	Fri Oct 16 00:08:00 2015 +0000
> : @@ -155,11 +155,17 @@ SKIP: {
> :
> : # verify that by default, 'combined' format is used, 'off' disables logging
> :
> : -$log = $t->read_file('combined.log');
> : -like($log,
> : -	qr!^\Q127.0.0.1 - - [\E .*
> : -		\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
> : -	'default log format');
> : +SKIP: {
> : +	my $sock = IO::Socket::INET->new(LocalAddr => '127.0.0.1');
> : +	skip(?127.0.0.1 local address required', 1) unless defined $sock;
>
> This looks like an excessive step.
> If perl could not successfully bind to 127.0.0.1 (or whatever it thinks to),
> nginx won?t start as well with failure effectively skipping all the tests.
>
> : +
> : +	$log = $t->read_file('combined.log');
> : +	my $addr = $sock->sockhost;
>
> I suggest to use a method with explicit parentheses,
> also for consistency with the rest nginx test suite.
>
> : +	like($log,
> : +		qr!^\Q$addr - - [\E .*
> : +			\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
> : +		'default log format');
> : +}
> :
> : # verify that log filtering works
> :
>
> I intend to commit this change:
>
> # HG changeset patch
> # User Sergey Kandaurov <pluknet at nginx.com>
> # Date 1445289338 -10800
> #      Tue Oct 20 00:15:38 2015 +0300
> # Node ID 0af5555e0b7b130d7162d304b074932331267882
> # Parent  cdd3659d114487d55119e1bb45f9a5fb44dbf79c
> Tests: unbreak access_log tests in jails.
>
> See ea574aae77cb for details.
>
> Reported by Steven Hartland, see
> http://mailman.nginx.org/pipermail/nginx-devel/2015-October/007393.html.
>
> diff -r cdd3659d1144 -r 0af5555e0b7b access_log.t
> --- a/access_log.t	Mon Oct 19 20:43:22 2015 +0300
> +++ b/access_log.t	Tue Oct 20 00:15:38 2015 +0300
> @@ -155,9 +155,11 @@ SKIP: {
>   
>   # verify that by default, 'combined' format is used, 'off' disables logging
>   
> +my $addr = IO::Socket::INET->new(LocalAddr => '127.0.0.1')->sockhost();
> +
>   $log = $t->read_file('combined.log');
>   like($log,
> -	qr!^\Q127.0.0.1 - - [\E .*
> +	qr!^\Q$addr - - [\E .*
>   		\Q] "GET /combined HTTP/1.0" 200 2 "-" "-"\E$!x,
>   	'default log format');
Looks good to me :)


From kou at cozmixng.org  Tue Oct 20 00:00:31 2015
From: kou at cozmixng.org (Kouhei Sutou)
Date: Tue, 20 Oct 2015 09:00:31 +0900 (JST)
Subject: [PATCH] Win32: make buildable with MinGW-w64 gcc
In-Reply-To: <20151019205435.GV48365@mdounin.ru>
References: <20151017.214432.472412871167194720.kou@cozmixng.org>
 <20151019205435.GV48365@mdounin.ru>
Message-ID: <20151020.090031.1431251694427611999.kou@cozmixng.org>

Hi,

In <20151019205435.GV48365 at mdounin.ru>
  "Re: [PATCH] Win32: make buildable with MinGW-w64 gcc" on Mon, 19 Oct 2015 23:54:35 +0300,
  Maxim Dounin <mdounin at mdounin.ru> wrote:

>> The attached patch fixes the following warning:
>> 
>>     x86_64-w64-mingw32-gcc -c -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/win32 -I objs \
>>             -o objs/src/core/ngx_connection.o \
>>             src/core/ngx_connection.c
>>     src/core/ngx_connection.c: In function 'ngx_close_idle_connections':
>>     src/core/ngx_connection.c:1178:21: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
>>              if (c[i].fd != -1 && c[i].idle) {
>>                          ^
>>     cc1: all warnings being treated as errors
> 
> [...]
> 
> Thanks, and this manifests itself with MinGW gcc as well.  
> Committed with some changes to commit log:
> 
> http://hg.nginx.org/nginx/rev/954b67727af3

Thanks for improving my patch!!!

--
kou


From mdounin at mdounin.ru  Tue Oct 20 01:52:10 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 20 Oct 2015 04:52:10 +0300
Subject: [PATCH] Support FreeBSD jails for testing
In-Reply-To: <5625711F.208@multiplay.co.uk>
References: <c22d8299e7040e0de6f8.1444954189@steve-dev.dev.multiplay.local>
 <20151016120537.GE48365@mdounin.ru>
 <562132BB.3030709@multiplay.co.uk>
 <20151019175727.GU48365@mdounin.ru> <5625711F.208@multiplay.co.uk>
Message-ID: <20151020015210.GW48365@mdounin.ru>

Hello!

On Mon, Oct 19, 2015 at 11:39:27PM +0100, Steven Hartland wrote:

[...]

> >>That doesn't fix the directory permission issue which causes pretty much
> >>every test to fail, so is this still an option for inclusion?
> >Directory premissions may vary depending on umask used.  If your
> >umask breaks tests for you - you may try changing it while running
> >tests.  It also shouldn't be important when running tests under
> >non-privileged user.
> Confirmed it doesn't effect a non-privileged user.
> 
> The default umask is 022 hence the issue if its run as privileged user with
> the default user as nobody which will have no access to the created
> directory.

Err, it looks like umask is unrelated here - it's File::Temp which 
creates temp directory with 0700 access mode.

[...]

> Something to mention in the README, if you don't like the chmod?

For sure we don't want chmod(0777), as an world-writable temp 
directory can easily result in problems when running tests on a 
host with untrusted users.  We can consider something like 
chmod(0755), but I'm not sure it's a good idea either, and this 
won't fix at least some tests.

As for the README, it already suggests to run under a 
non-privileged user - it uses "$ " shell prompt in the usage 
example.  ;)  If you think it's not enough and needs to be said 
explicitly - patches are welcome.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Tue Oct 20 02:00:47 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 20 Oct 2015 05:00:47 +0300
Subject: Error handling from filter modules
In-Reply-To: <5625688D.9030004@multiplay.co.uk>
References: <5620548D.8010601@multiplay.co.uk>
 <20151016122005.GG48365@mdounin.ru>
 <562130B2.10001@multiplay.co.uk>
 <20151019165940.GQ48365@mdounin.ru>
 <5625688D.9030004@multiplay.co.uk>
Message-ID: <20151020020047.GX48365@mdounin.ru>

Hello!

On Mon, Oct 19, 2015 at 11:02:53PM +0100, Steven Hartland wrote:

[...]

> >When the NGX_ERROR case is triggered, no response is returned, the
> >connection is just closed.
> It does seem like the connection to the client is just closed however a 206
> response is logged for the request in the access log, hence my confusion; is
> that expected?

Yes.  Once set, status code will be logged even if an error occurs 
later.

> >As this expected to happen if and only
> >if we weren't able to allocate memory, this is believed to be
> >perfectly correct behaviour.  Trying to do anything else is very
> >unlikely to succeed, but likely to result in additional problems.
> >
> In the current code yes, but in the patch that I was testing there's one
> additional case which can return an error. This was caused by an intentional
> configuration error (test designed to confirm the behaviour in this case).
> 
> Its likely going to be difficult to discuss fully without the patch to
> comment against so I'll finalise that and get it posted as soon as I can.

General recommendation is to avoid cases when you have to return  
error pages from filter modules.  But yes, it's difficult to 
disscuss without the code.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Tue Oct 20 15:18:12 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 20 Oct 2015 15:18:12 +0000
Subject: [nginx] Win32: timer_resolution now ignored with select.
Message-ID: <hg.dceb078b1156.1445354292.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/dceb078b1156
branches:  
changeset: 6266:dceb078b1156
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Oct 20 18:16:18 2015 +0300
description:
Win32: timer_resolution now ignored with select.

As setitimer() isn't available on Windows, time wasn't updated at all
if timer_resolution was used with the select event method.  Fix is
to ignore timer_resolution in such cases.

diffstat:

 src/event/ngx_event.c |  9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diffs (19 lines):

diff --git a/src/event/ngx_event.c b/src/event/ngx_event.c
--- a/src/event/ngx_event.c
+++ b/src/event/ngx_event.c
@@ -670,6 +670,15 @@ ngx_event_process_init(ngx_cycle_t *cycl
         }
     }
 
+#else
+
+    if (ngx_timer_resolution && !(ngx_event_flags & NGX_USE_TIMER_EVENT)) {
+        ngx_log_error(NGX_LOG_WARN, cycle->log, 0,
+                      "the \"timer_resolution\" directive is not supported "
+                      "with the configured event method, ignored");
+        ngx_timer_resolution = 0;
+    }
+
 #endif
 
     cycle->connections =


From vbart at nginx.com  Tue Oct 20 18:33:01 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Tue, 20 Oct 2015 18:33:01 +0000
Subject: [nginx] HTTP/2: improved HPACK integer parsing code readability.
Message-ID: <hg.adaac65899c8.1445365981.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/adaac65899c8
branches:  
changeset: 6267:adaac65899c8
user:      Ruslan Ermilov <ru at nginx.com>
date:      Tue Oct 20 21:26:38 2015 +0300
description:
HTTP/2: improved HPACK integer parsing code readability.

No functional changes.

diffstat:

 src/http/v2/ngx_http_v2.c |  6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diffs (23 lines):

diff -r dceb078b1156 -r adaac65899c8 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Tue Oct 20 18:16:18 2015 +0300
+++ b/src/http/v2/ngx_http_v2.c	Tue Oct 20 21:26:38 2015 +0300
@@ -2396,8 +2396,8 @@ ngx_http_v2_parse_int(ngx_http_v2_connec
         return value;
     }
 
-    if (end - p > NGX_HTTP_V2_INT_OCTETS - 1) {
-        end = p + NGX_HTTP_V2_INT_OCTETS - 1;
+    if (end - start > NGX_HTTP_V2_INT_OCTETS) {
+        end = start + NGX_HTTP_V2_INT_OCTETS;
     }
 
     for (shift = 0; p != end; shift += 7) {
@@ -2417,7 +2417,7 @@ ngx_http_v2_parse_int(ngx_http_v2_connec
         }
     }
 
-    if ((size_t) (end - start) >= NGX_HTTP_V2_INT_OCTETS) {
+    if (end == start + NGX_HTTP_V2_INT_OCTETS) {
         return NGX_DECLINED;
     }
 


From vbart at nginx.com  Tue Oct 20 18:33:03 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Tue, 20 Oct 2015 18:33:03 +0000
Subject: [nginx] HTTP/2: improved error handling while parsing integers.
Message-ID: <hg.ee16fb0db905.1445365983.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/ee16fb0db905
branches:  
changeset: 6268:ee16fb0db905
user:      Valentin Bartenev <vbart at nginx.com>
date:      Tue Oct 20 21:28:38 2015 +0300
description:
HTTP/2: improved error handling while parsing integers.

The case when an integer is out of frame bounds should be checked first
as a more accurate error.

diffstat:

 src/http/v2/ngx_http_v2.c |  8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diffs (22 lines):

diff -r adaac65899c8 -r ee16fb0db905 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Tue Oct 20 21:26:38 2015 +0300
+++ b/src/http/v2/ngx_http_v2.c	Tue Oct 20 21:28:38 2015 +0300
@@ -2417,14 +2417,14 @@ ngx_http_v2_parse_int(ngx_http_v2_connec
         }
     }
 
+    if ((size_t) (end - start) >= h2c->state.length) {
+        return NGX_ERROR;
+    }
+
     if (end == start + NGX_HTTP_V2_INT_OCTETS) {
         return NGX_DECLINED;
     }
 
-    if ((size_t) (end - start) >= h2c->state.length) {
-        return NGX_ERROR;
-    }
-
     return NGX_AGAIN;
 }
 


From mhenrion at appnexus.com  Thu Oct 22 12:30:54 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Thu, 22 Oct 2015 12:30:54 +0000
Subject: Processing responses of unbounded sizes from upstream servers
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>

Hello all,


I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.

It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the upstream structure and I haven't been able to locate a piece of code that would do things differently.

I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?

Thanks a lot in advance!
Maxime

From 3tnica at gmail.com  Fri Oct 23 18:51:56 2015
From: 3tnica at gmail.com (Nils Hermansson)
Date: Fri, 23 Oct 2015 20:51:56 +0200
Subject: Syslog Unix socket patch
Message-ID: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>

# HG changeset patch
# User Nils Hermansson <3tnica at gmail.com>
# Date 1445625283 -7200
#      Fri Oct 23 20:34:43 2015 +0200
# Node ID 868fc6b3bf69be611118c597578e749c65698b8c
# Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
Most standard syslog facilicties do not expect hostname when logging to
Unix Sockets.
This patch removes hostname from syslog message when logging to Unix Socket.
Tested on rsyslog and syslog-ng

diff -r ee16fb0db905 -r 868fc6b3bf69 src/core/ngx_syslog.c
--- a/src/core/ngx_syslog.c     Tue Oct 20 21:28:38 2015 +0300
+++ b/src/core/ngx_syslog.c     Fri Oct 23 20:34:43 2015 +0200
@@ -219,9 +219,17 @@
     ngx_uint_t  pri;

     pri = peer->facility * 8 + peer->severity;
-
-    return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
&ngx_cached_syslog_time,
+#if (NGX_HAVE_UNIX_DOMAIN)
+    if  (peer->server.sockaddr->sa_family == AF_UNIX) {
+        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
&ngx_cached_syslog_time,
+                       &peer->tag);
+    } else {
+#endif
+        return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
&ngx_cached_syslog_time,
                        &ngx_cycle->hostname, &peer->tag);
+#if (NGX_HAVE_UNIX_DOMAIN)
+    }
+#endif
 }
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151023/37b97860/attachment.html>

From vincent.schwarzer at yahoo.de  Sat Oct 24 14:25:22 2015
From: vincent.schwarzer at yahoo.de (vincent.schwarzer at yahoo.de)
Date: Sat, 24 Oct 2015 14:25:22 +0000 (UTC)
Subject: Compile NGINX as Position Independent Executable (Compile Error)
References: <2054122694.3616875.1445696722263.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <2054122694.3616875.1445696722263.JavaMail.yahoo@mail.yahoo.com>

Hey NGINX Devs,
I want to compile and run NGINX on OSv from Cloudius?https://github.com/ cloudius-systems/osv.For that I have to compile NGINX as Position Independent Code (-fPIC / -fPIE).

The issue I currently have is that when I run 'make build' after executing the ./configure script in the source folder I get following error:

cc -c -pipe ?-O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -fPIE -I src/core -I src/event -I src/event/modules -I src/os/unix -I objs ?-o objs/src/os/unix/ngx_process_ cycle.o \ src/os/unix/ngx_process_cycle. o src/os/unix/ngx_process_cycle. c: In function ?ngx_master_process_cycle?:

src/os/unix/ngx_process_cycle. c:124:5: error: implicit declaration of function ?setproctitle? [-Werror=implicit-function- declaration]
ngx_setproctitle(title);
? ? ?^
cc1: all warnings being treated as errors
make[1]: *** [objs/src/os/unix/ngx_process_ cycle.o] Error 1
make[1]: Leaving directory `/tmp/thesis_vs/osv_proj/ nginx/upstream/nginx-1.8.0'
make: *** [build] Error 2



My current workflow to compile NGNIX with fPIE is as follows:
   
   - Load Sources (1.8 Stable Release)
   - Patch source files from?http://git.buildroot.net/ buildroot/tree/package/nginx?to pass some of the configure checks and adjust as needed for OSv Operating System.
   - Export flags for configure script as required for OSv   



export ngx_force_gcc_have_atomic=yes \

export ngx_force_c99_have_variadic_ macros=yes \

export ngx_force_gcc_have_variadic_ macros=yes \

export ngx_force_have_epoll=yes \

export ngx_force_have_sendfile=yes \

export ngx_force_have_sendfile64=yes \ ?!??!

export ngx_force_have_pr_set_ dumpable=yes \ ?!?!

export ngx_force_sys_nerr=100 \

export ngx_force_have_map_anon=yes \

export ngx_force_have_posix_sem=yes\

export ngx_force_have_sysvshm=yes\

export ngx_force_have_map_devzero=no




4.?./configure --with-cc-opt="-fPIE" \

--with-ld-opt="-shared" \
--prefix=/none \
--conf-path=/data/conf/nginx. conf \
--sbin-path=/none \
--pid-path=/tmp/nginx.pid \
--lock-path=/tmp/nginx.lock \
--error-log-path=/tmp/error. log \
--http-log-path=/tmp/access. log \
--http-client-body-temp-path=/ tmp/client-body \
--http-proxy-temp-path=/tmp/ proxy \
--http-fastcgi-temp-path=/tmp/ fastcgi \
--http-scgi-temp-path=/tmp/ scgi \
--http-uwsgi-temp-path=/tmp/ uwsgi \
--without-http_rewrite_module \
--with-http_ssl_module \
--with-ipv6 \
--with-pcre?

5. make build


Im using Ubuntu 14.04 and gcc-4.8 to compile NGINX
I have no idea if / what I'm doing wrong here. Any help is highly appreciated. :)?
Kind regards,
Vincent
P.S.: Sorry for sending it on both lists but I wasn't sure which is the right one for this kind of question.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151024/840704d6/attachment.html>

From piotrsikora at google.com  Sat Oct 24 23:26:09 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:09 -0700
Subject: [PATCH] Remove src/core/ngx_crc.h, unused since nginx-0.1.13
Message-ID: <8c25beef3cc4f7d3b88c.1445729169@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 8c25beef3cc4f7d3b88c8718bd24f02b1bc39d35
# Parent  a8be1943bde3ea151ade97149856684db4066fe3
Remove src/core/ngx_crc.h, unused since nginx-0.1.13.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r a8be1943bde3 -r 8c25beef3cc4 auto/sources
--- a/auto/sources
+++ b/auto/sources
@@ -22,7 +22,6 @@ CORE_DEPS="src/core/nginx.h \
            src/core/ngx_parse_time.h \
            src/core/ngx_inet.h \
            src/core/ngx_file.h \
-           src/core/ngx_crc.h \
            src/core/ngx_crc32.h \
            src/core/ngx_murmurhash.h \
            src/core/ngx_md5.h \
diff -r a8be1943bde3 -r 8c25beef3cc4 src/core/ngx_core.h
--- a/src/core/ngx_core.h
+++ b/src/core/ngx_core.h
@@ -64,7 +64,6 @@ typedef void (*ngx_connection_handler_pt
 #include <ngx_list.h>
 #include <ngx_hash.h>
 #include <ngx_file.h>
-#include <ngx_crc.h>
 #include <ngx_crc32.h>
 #include <ngx_murmurhash.h>
 #if (NGX_PCRE)
diff -r a8be1943bde3 -r 8c25beef3cc4 src/core/ngx_crc.h
--- a/src/core/ngx_crc.h
+++ /dev/null
@@ -1,39 +0,0 @@
-
-/*
- * Copyright (C) Igor Sysoev
- * Copyright (C) Nginx, Inc.
- */
-
-
-#ifndef _NGX_CRC_H_INCLUDED_
-#define _NGX_CRC_H_INCLUDED_
-
-
-#include <ngx_config.h>
-#include <ngx_core.h>
-
-
-/* 32-bit crc16 */
-
-static ngx_inline uint32_t
-ngx_crc(u_char *data, size_t len)
-{
-    uint32_t  sum;
-
-    for (sum = 0; len; len--) {
-
-        /*
-         * gcc 2.95.2 x86 and icc 7.1.006 compile
-         * that operator into the single "rol" opcode,
-         * msvc 6.0sp2 compiles it into four opcodes.
-         */
-        sum = sum >> 1 | sum << 31;
-
-        sum += *data++;
-    }
-
-    return sum;
-}
-
-
-#endif /* _NGX_CRC_H_INCLUDED_ */


From piotrsikora at google.com  Sat Oct 24 23:26:04 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:04 -0700
Subject: [PATCH] Remove auto/lib/test, unused since nginx-0.1.2
Message-ID: <a8be1943bde3ea151ade.1445729164@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID a8be1943bde3ea151ade97149856684db4066fe3
# Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
Remove auto/lib/test, unused since nginx-0.1.2.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r ee16fb0db905 -r a8be1943bde3 auto/lib/test
--- a/auto/lib/test
+++ /dev/null
@@ -1,40 +0,0 @@
-
-# Copyright (C) Igor Sysoev
-# Copyright (C) Nginx, Inc.
-
-
-echo $ngx_n "checking for $ngx_lib ...$ngx_c"
-
-cat << END >> $NGX_AUTOCONF_ERR
-
-----------------------------------------
-checking for $ngx_lib
-
-END
-
-ngx_found=no
-
-cat << END > $NGX_AUTOTEST.c
-
-$ngx_lib_incs
-
-int main() {
-    $ngx_lib_test;
-    return 0;
-}
-
-
-eval "$CC $cc_test_flags $ngx_lib_cflags \
-     -o $NGX_AUTOTEST $NGX_AUTOTEST.c $ngx_libs \
-     >> $NGX_ERR 2>&1"
-
-if [ -x $NGX_AUTOTEST ]; then
-    echo " found"
-
-    ngx_found=yes
-
-else
-    echo " not found"
-fi
-
-rm -rf $NGX_AUTOTEST*


From piotrsikora at google.com  Sat Oct 24 23:26:16 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:16 -0700
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL
Message-ID: <7435401242d6dbfdb4b4.1445729176@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 7435401242d6dbfdb4b42339ab2d96ed30d397a7
# Parent  8c25beef3cc4f7d3b88c8718bd24f02b1bc39d35
Configure: remove redundant NGX_OPENSSL.

Previously, both NGX_OPENSSL (used to init library and include headers)
and NGX_SSL (used in rest of the source code) had to be defined.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r 8c25beef3cc4 -r 7435401242d6 auto/lib/openssl/conf
--- a/auto/lib/openssl/conf
+++ b/auto/lib/openssl/conf
@@ -8,7 +8,6 @@ if [ $OPENSSL != NONE ]; then
     case "$CC" in
 
         cl | bcc32)
-            have=NGX_OPENSSL . auto/have
             have=NGX_SSL . auto/have
 
             CFLAGS="$CFLAGS -DNO_SYS_TYPES_H"
@@ -25,7 +24,6 @@ if [ $OPENSSL != NONE ]; then
         ;;
 
         *)
-            have=NGX_OPENSSL . auto/have
             have=NGX_SSL . auto/have
 
             CORE_INCS="$CORE_INCS $OPENSSL/.openssl/include"
@@ -47,7 +45,7 @@ else
         OPENSSL=NO
 
         ngx_feature="OpenSSL library"
-        ngx_feature_name="NGX_OPENSSL"
+        ngx_feature_name="NGX_SSL"
         ngx_feature_run=no
         ngx_feature_incs="#include <openssl/ssl.h>"
         ngx_feature_path=
@@ -104,7 +102,6 @@ else
         fi
 
         if [ $ngx_found = yes ]; then
-            have=NGX_SSL . auto/have
             CORE_LIBS="$CORE_LIBS $ngx_feature_libs $NGX_LIBDL"
             OPENSSL=YES
         fi
diff -r 8c25beef3cc4 -r 7435401242d6 src/core/nginx.c
--- a/src/core/nginx.c
+++ b/src/core/nginx.c
@@ -218,7 +218,7 @@ main(int argc, char *const *argv)
     }
 
     /* STUB */
-#if (NGX_OPENSSL)
+#if (NGX_SSL)
     ngx_ssl_init(log);
 #endif
 
diff -r 8c25beef3cc4 -r 7435401242d6 src/core/ngx_core.h
--- a/src/core/ngx_core.h
+++ b/src/core/ngx_core.h
@@ -77,7 +77,7 @@ typedef void (*ngx_connection_handler_pt
 #include <ngx_inet.h>
 #include <ngx_cycle.h>
 #include <ngx_resolver.h>
-#if (NGX_OPENSSL)
+#if (NGX_SSL)
 #include <ngx_event_openssl.h>
 #endif
 #include <ngx_process_cycle.h>


From piotrsikora at google.com  Sat Oct 24 23:26:22 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:22 -0700
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL_MD5
Message-ID: <84b5500257121c6bd3e1.1445729182@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 84b5500257121c6bd3e1c0e778cfde8915bf3fcc
# Parent  7435401242d6dbfdb4b42339ab2d96ed30d397a7
Configure: remove redundant NGX_OPENSSL_MD5.

Previously, both NGX_OPENSSL_MD5 and NGX_HAVE_OPENSSL_MD5_H
had to be defined.

While there, add missing _H to NGX_HAVE_OPENSSL_MD5.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r 7435401242d6 -r 84b550025712 auto/lib/conf
--- a/auto/lib/conf
+++ b/auto/lib/conf
@@ -29,7 +29,6 @@ if [ $USE_MD5 = YES ]; then
 
     if [ $USE_OPENSSL = YES ]; then
         have=NGX_HAVE_OPENSSL_MD5_H . auto/have
-        have=NGX_OPENSSL_MD5 . auto/have
         have=NGX_HAVE_MD5 . auto/have
         MD5=YES
         MD5_LIB=OpenSSL
diff -r 7435401242d6 -r 84b550025712 auto/lib/md5/conf
--- a/auto/lib/md5/conf
+++ b/auto/lib/md5/conf
@@ -8,8 +8,7 @@ if [ $MD5 != NONE ]; then
     if grep MD5_Init $MD5/md5.h 2>&1 >/dev/null; then
         # OpenSSL md5
         OPENSSL_MD5=YES
-        have=NGX_HAVE_OPENSSL_MD5 . auto/have
-        have=NGX_OPENSSL_MD5 . auto/have
+        have=NGX_HAVE_OPENSSL_MD5_H . auto/have
     else
         # rsaref md5
         OPENSSL_MD5=NO
@@ -79,7 +78,6 @@ else
             # OpenSSL crypto library
 
             ngx_feature="md5 in system OpenSSL crypto library"
-            ngx_feature_name="NGX_OPENSSL_MD5"
             ngx_feature_incs="#include <openssl/md5.h>"
             ngx_feature_libs="-lcrypto"
             ngx_feature_test="MD5_CTX md5; MD5_Init(&md5)"
@@ -89,7 +87,6 @@ else
 
             if [ $ngx_found = yes ]; then
                 have=NGX_HAVE_OPENSSL_MD5_H . auto/have
-                have=NGX_HAVE_MD5 . auto/have
             fi
         fi
 
diff -r 7435401242d6 -r 84b550025712 src/core/ngx_md5.h
--- a/src/core/ngx_md5.h
+++ b/src/core/ngx_md5.h
@@ -25,7 +25,7 @@
 typedef MD5_CTX  ngx_md5_t;
 
 
-#if (NGX_OPENSSL_MD5)
+#if (NGX_HAVE_OPENSSL_MD5_H)
 
 #define ngx_md5_init    MD5_Init
 #define ngx_md5_update  MD5_Update


From piotrsikora at google.com  Sat Oct 24 23:26:26 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:26 -0700
Subject: [PATCH] Configure: style fixes for autoconf.err
Message-ID: <960c335482a516878355.1445729186@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 960c335482a516878355bd42b0c8cf58aa112703
# Parent  84b5500257121c6bd3e1c0e778cfde8915bf3fcc
Configure: style fixes for autoconf.err.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r 84b550025712 -r 960c335482a5 auto/endianness
--- a/auto/endianness
+++ b/auto/endianness
@@ -4,8 +4,13 @@
 
 
 echo $ngx_n "checking for system byte ordering ...$ngx_c"
-echo >> $NGX_ERR
-echo "checking for system byte ordering" >> $NGX_ERR
+
+cat << END >> $NGX_AUTOCONF_ERR
+
+----------------------------------------
+checking for system byte ordering
+
+END
 
 
 cat << END > $NGX_AUTOTEST.c
diff -r 84b550025712 -r 960c335482a5 auto/types/uintptr_t
--- a/auto/types/uintptr_t
+++ b/auto/types/uintptr_t
@@ -4,8 +4,13 @@
 
 
 echo $ngx_n "checking for uintptr_t ...$ngx_c"
-echo >> $NGX_AUTOCONF_ERR
-echo "checking for uintptr_t" >> $NGX_AUTOCONF_ERR
+
+cat << END >> $NGX_AUTOCONF_ERR
+
+----------------------------------------
+checking for uintptr_t
+
+END
 
 found=no
 


From piotrsikora at google.com  Sat Oct 24 23:26:35 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:35 -0700
Subject: [PATCH] HTTP: include SSI header only where it's needed
Message-ID: <685e87ae457b29e30ca9.1445729195@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 685e87ae457b29e30ca99e1ae039f4a87ccc1b8e
# Parent  903e6a14b17e9140ee88f671f1ba2293c512ab3b
HTTP: include SSI header only where it's needed.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r 903e6a14b17e -r 685e87ae457b src/http/modules/ngx_http_ssi_filter_module.c
--- a/src/http/modules/ngx_http_ssi_filter_module.c
+++ b/src/http/modules/ngx_http_ssi_filter_module.c
@@ -8,6 +8,7 @@
 #include <ngx_config.h>
 #include <ngx_core.h>
 #include <ngx_http.h>
+#include <ngx_http_ssi_filter_module.h>
 
 #define NGX_HTTP_SSI_ERROR          1
 
diff -r 903e6a14b17e -r 685e87ae457b src/http/modules/perl/ngx_http_perl_module.h
--- a/src/http/modules/perl/ngx_http_perl_module.h
+++ b/src/http/modules/perl/ngx_http_perl_module.h
@@ -14,6 +14,10 @@
 #include <ngx_http.h>
 #include <nginx.h>
 
+#if (NGX_HTTP_SSI)
+#include <ngx_http_ssi_filter_module.h>
+#endif
+
 #include <EXTERN.h>
 #include <perl.h>
 
diff -r 903e6a14b17e -r 685e87ae457b src/http/ngx_http.h
--- a/src/http/ngx_http.h
+++ b/src/http/ngx_http.h
@@ -44,9 +44,6 @@ typedef u_char *(*ngx_http_log_handler_p
 #if (NGX_HTTP_CACHE)
 #include <ngx_http_cache.h>
 #endif
-#if (NGX_HTTP_SSI)
-#include <ngx_http_ssi_filter_module.h>
-#endif
 #if (NGX_HTTP_SSL)
 #include <ngx_http_ssl_module.h>
 #endif


From piotrsikora at google.com  Sat Oct 24 23:26:30 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Sat, 24 Oct 2015 16:26:30 -0700
Subject: [PATCH] Core: read/write locks are also required by the Stream module
Message-ID: <903e6a14b17e9140ee88.1445729190@piotrsikora.sfo.corp.google.com>

# HG changeset patch
# User Piotr Sikora <piotrsikora at google.com>
# Date 1445649693 25200
#      Fri Oct 23 18:21:33 2015 -0700
# Node ID 903e6a14b17e9140ee88f671f1ba2293c512ab3b
# Parent  960c335482a516878355bd42b0c8cf58aa112703
Core: read/write locks are also required by the Stream module.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diff -r 960c335482a5 -r 903e6a14b17e src/core/ngx_rwlock.c
--- a/src/core/ngx_rwlock.c
+++ b/src/core/ngx_rwlock.c
@@ -111,7 +111,7 @@ ngx_rwlock_unlock(ngx_atomic_t *lock)
 
 #else
 
-#if (NGX_HTTP_UPSTREAM_ZONE)
+#if (NGX_HTTP_UPSTREAM_ZONE || NGX_STREAM_UPSTREAM_ZONE)
 
 #error ngx_atomic_cmp_set() is not defined!
 


From ru at nginx.com  Mon Oct 26 11:18:36 2015
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 26 Oct 2015 11:18:36 +0000
Subject: [nginx] Configure: style fixes for autoconf.err.
Message-ID: <hg.7ec809b579d7.1445858316.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/7ec809b579d7
branches:  
changeset: 6269:7ec809b579d7
user:      Piotr Sikora <piotrsikora at google.com>
date:      Fri Oct 23 18:21:33 2015 -0700
description:
Configure: style fixes for autoconf.err.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diffstat:

 auto/endianness      |  9 +++++++--
 auto/types/uintptr_t |  9 +++++++--
 2 files changed, 14 insertions(+), 4 deletions(-)

diffs (38 lines):

diff -r ee16fb0db905 -r 7ec809b579d7 auto/endianness
--- a/auto/endianness	Tue Oct 20 21:28:38 2015 +0300
+++ b/auto/endianness	Fri Oct 23 18:21:33 2015 -0700
@@ -4,8 +4,13 @@
 
 
 echo $ngx_n "checking for system byte ordering ...$ngx_c"
-echo >> $NGX_ERR
-echo "checking for system byte ordering" >> $NGX_ERR
+
+cat << END >> $NGX_AUTOCONF_ERR
+
+----------------------------------------
+checking for system byte ordering
+
+END
 
 
 cat << END > $NGX_AUTOTEST.c
diff -r ee16fb0db905 -r 7ec809b579d7 auto/types/uintptr_t
--- a/auto/types/uintptr_t	Tue Oct 20 21:28:38 2015 +0300
+++ b/auto/types/uintptr_t	Fri Oct 23 18:21:33 2015 -0700
@@ -4,8 +4,13 @@
 
 
 echo $ngx_n "checking for uintptr_t ...$ngx_c"
-echo >> $NGX_AUTOCONF_ERR
-echo "checking for uintptr_t" >> $NGX_AUTOCONF_ERR
+
+cat << END >> $NGX_AUTOCONF_ERR
+
+----------------------------------------
+checking for uintptr_t
+
+END
 
 found=no
 


From ru at nginx.com  Mon Oct 26 11:18:38 2015
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 26 Oct 2015 11:18:38 +0000
Subject: [nginx] Core: read/write locks are also required by the Stream m...
Message-ID: <hg.e769cc88f996.1445858318.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/e769cc88f996
branches:  
changeset: 6270:e769cc88f996
user:      Piotr Sikora <piotrsikora at google.com>
date:      Fri Oct 23 18:21:33 2015 -0700
description:
Core: read/write locks are also required by the Stream module.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diffstat:

 src/core/ngx_rwlock.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 7ec809b579d7 -r e769cc88f996 src/core/ngx_rwlock.c
--- a/src/core/ngx_rwlock.c	Fri Oct 23 18:21:33 2015 -0700
+++ b/src/core/ngx_rwlock.c	Fri Oct 23 18:21:33 2015 -0700
@@ -111,7 +111,7 @@ ngx_rwlock_unlock(ngx_atomic_t *lock)
 
 #else
 
-#if (NGX_HTTP_UPSTREAM_ZONE)
+#if (NGX_HTTP_UPSTREAM_ZONE || NGX_STREAM_UPSTREAM_ZONE)
 
 #error ngx_atomic_cmp_set() is not defined!
 


From ru at nginx.com  Mon Oct 26 11:19:37 2015
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 26 Oct 2015 14:19:37 +0300
Subject: [PATCH] Configure: style fixes for autoconf.err
In-Reply-To: <960c335482a516878355.1445729186@piotrsikora.sfo.corp.google.com>
References: <960c335482a516878355.1445729186@piotrsikora.sfo.corp.google.com>
Message-ID: <20151026111937.GB5544@lo0.su>

On Sat, Oct 24, 2015 at 04:26:26PM -0700, Piotr Sikora wrote:
> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1445649693 25200
> #      Fri Oct 23 18:21:33 2015 -0700
> # Node ID 960c335482a516878355bd42b0c8cf58aa112703
> # Parent  84b5500257121c6bd3e1c0e778cfde8915bf3fcc
> Configure: style fixes for autoconf.err.
> 
> Signed-off-by: Piotr Sikora <piotrsikora at google.com>

Committed, thanks!


From ru at nginx.com  Mon Oct 26 11:19:56 2015
From: ru at nginx.com (Ruslan Ermilov)
Date: Mon, 26 Oct 2015 14:19:56 +0300
Subject: [PATCH] Core: read/write locks are also required by the Stream
 module
In-Reply-To: <903e6a14b17e9140ee88.1445729190@piotrsikora.sfo.corp.google.com>
References: <903e6a14b17e9140ee88.1445729190@piotrsikora.sfo.corp.google.com>
Message-ID: <20151026111956.GC5544@lo0.su>

On Sat, Oct 24, 2015 at 04:26:30PM -0700, Piotr Sikora wrote:
> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1445649693 25200
> #      Fri Oct 23 18:21:33 2015 -0700
> # Node ID 903e6a14b17e9140ee88f671f1ba2293c512ab3b
> # Parent  960c335482a516878355bd42b0c8cf58aa112703
> Core: read/write locks are also required by the Stream module.
> 
> Signed-off-by: Piotr Sikora <piotrsikora at google.com>

Committed, thanks!


From mdounin at mdounin.ru  Mon Oct 26 12:50:25 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 26 Oct 2015 15:50:25 +0300
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL_MD5
In-Reply-To: <84b5500257121c6bd3e1.1445729182@piotrsikora.sfo.corp.google.com>
References: <84b5500257121c6bd3e1.1445729182@piotrsikora.sfo.corp.google.com>
Message-ID: <20151026125025.GS48365@mdounin.ru>

Hello!

On Sat, Oct 24, 2015 at 04:26:22PM -0700, Piotr Sikora wrote:

> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1445649693 25200
> #      Fri Oct 23 18:21:33 2015 -0700
> # Node ID 84b5500257121c6bd3e1c0e778cfde8915bf3fcc
> # Parent  7435401242d6dbfdb4b42339ab2d96ed30d397a7
> Configure: remove redundant NGX_OPENSSL_MD5.
> 
> Previously, both NGX_OPENSSL_MD5 and NGX_HAVE_OPENSSL_MD5_H
> had to be defined.
> 
> While there, add missing _H to NGX_HAVE_OPENSSL_MD5.

These macros are destinct.  The NGX_HAVE_OPENSSL_MD5_H means that 
we are using "openssl/md5.h" include.  Second one means that we 
are using OpenSSL-style function names, like "MD5_Init", and not 
RSA-like names like "MD5Init".

Such a distinction may be needed when using external MD5 library 
with --with-md5.

[...]

> diff -r 7435401242d6 -r 84b550025712 auto/lib/md5/conf
> --- a/auto/lib/md5/conf
> +++ b/auto/lib/md5/conf
> @@ -8,8 +8,7 @@ if [ $MD5 != NONE ]; then
>      if grep MD5_Init $MD5/md5.h 2>&1 >/dev/null; then
>          # OpenSSL md5
>          OPENSSL_MD5=YES
> -        have=NGX_HAVE_OPENSSL_MD5 . auto/have
> -        have=NGX_OPENSSL_MD5 . auto/have
> +        have=NGX_HAVE_OPENSSL_MD5_H . auto/have

I don't think this is a typo, rather it tries to define the 
NGX_HAVE_OPENSSL_MD5 macro, currently unused.  Note that it tests 
"md5.h", not "openssl/md5.h".

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Mon Oct 26 12:58:25 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 26 Oct 2015 15:58:25 +0300
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL
In-Reply-To: <7435401242d6dbfdb4b4.1445729176@piotrsikora.sfo.corp.google.com>
References: <7435401242d6dbfdb4b4.1445729176@piotrsikora.sfo.corp.google.com>
Message-ID: <20151026125825.GU48365@mdounin.ru>

Hello!

On Sat, Oct 24, 2015 at 04:26:16PM -0700, Piotr Sikora wrote:

> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1445649693 25200
> #      Fri Oct 23 18:21:33 2015 -0700
> # Node ID 7435401242d6dbfdb4b42339ab2d96ed30d397a7
> # Parent  8c25beef3cc4f7d3b88c8718bd24f02b1bc39d35
> Configure: remove redundant NGX_OPENSSL.
> 
> Previously, both NGX_OPENSSL (used to init library and include headers)
> and NGX_SSL (used in rest of the source code) had to be defined.

The NGX_SSL and NGX_OPENSSL macros are distinct as a result of an 
attempt to allow using different SSL libraries by introducing an 
abstraction ngx_ssl_* layer.  I don't think removing this 
distinction is a good idea.

[...]

-- 
Maxim Dounin
http://nginx.org/


From vbart at nginx.com  Mon Oct 26 14:39:28 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:39:28 +0000
Subject: [nginx] HTTP/2: reject self-dependent streams.
Message-ID: <hg.22ee99422329.1445870368.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/22ee99422329
branches:  
changeset: 6271:22ee99422329
user:      Piotr Sikora <piotrsikora at google.com>
date:      Thu Oct 01 20:25:55 2015 -0700
description:
HTTP/2: reject self-dependent streams.

Per RFC7540, a stream cannot depend on itself.

Previously, this requirement was enforced on PRIORITY frames, but not on
HEADERS frames and due to the implementation details nginx worker would
crash (stack overflow) while opening self-dependent stream.

Found with afl-fuzz.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diffstat:

 src/http/v2/ngx_http_v2.c |  16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diffs (26 lines):

diff -r e769cc88f996 -r 22ee99422329 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Fri Oct 23 18:21:33 2015 -0700
+++ b/src/http/v2/ngx_http_v2.c	Thu Oct 01 20:25:55 2015 -0700
@@ -1133,6 +1133,22 @@ ngx_http_v2_state_headers(ngx_http_v2_co
 
     h2c->last_sid = h2c->state.sid;
 
+    if (depend == h2c->state.sid) {
+        ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
+                      "client sent HEADERS frame for stream %ui "
+                      "with incorrect dependency", h2c->state.sid);
+
+        if (ngx_http_v2_send_rst_stream(h2c, h2c->state.sid,
+                                        NGX_HTTP_V2_PROTOCOL_ERROR)
+            != NGX_OK)
+        {
+            return ngx_http_v2_connection_error(h2c,
+                                                NGX_HTTP_V2_INTERNAL_ERROR);
+        }
+
+        return ngx_http_v2_state_skip_headers(h2c, pos, end);
+    }
+
     h2scf = ngx_http_get_module_srv_conf(h2c->http_connection->conf_ctx,
                                          ngx_http_v2_module);
 


From vbart at nginx.com  Mon Oct 26 14:39:30 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:39:30 +0000
Subject: [nginx] HTTP/2: fix indirect reprioritization.
Message-ID: <hg.b6a665bf858a.1445870370.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b6a665bf858a
branches:  
changeset: 6272:b6a665bf858a
user:      Piotr Sikora <piotrsikora at google.com>
date:      Thu Oct 01 20:25:55 2015 -0700
description:
HTTP/2: fix indirect reprioritization.

Previously, streams that were indirectly reprioritized (either because of
a new exclusive dependency on their parent or because of removal of their
parent from the dependency tree), didn't have their pointer to the parent
node updated.

This broke detection of circular dependencies and, as a result, nginx
worker would crash due to stack overflow whenever such dependency was
introduced.

Found with afl-fuzz.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diffstat:

 src/http/v2/ngx_http_v2.c |  17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diffs (53 lines):

diff -r 22ee99422329 -r b6a665bf858a src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Thu Oct 01 20:25:55 2015 -0700
+++ b/src/http/v2/ngx_http_v2.c	Thu Oct 01 20:25:55 2015 -0700
@@ -2914,11 +2914,14 @@ ngx_http_v2_get_closed_node(ngx_http_v2_
         weight += child->weight;
     }
 
+    parent = node->parent;
+
     for (q = ngx_queue_head(&node->children);
          q != ngx_queue_sentinel(&node->children);
          q = ngx_queue_next(q))
     {
         child = ngx_queue_data(q, ngx_http_v2_node_t, queue);
+        child->parent = parent;
         child->weight = node->weight * child->weight / weight;
 
         if (child->weight == 0) {
@@ -2926,8 +2929,6 @@ ngx_http_v2_get_closed_node(ngx_http_v2_
         }
     }
 
-    parent = node->parent;
-
     if (parent == NGX_HTTP_V2_ROOT) {
         node->rank = 0;
         node->rel_weight = 1.0;
@@ -3940,8 +3941,8 @@ static void
 ngx_http_v2_set_dependency(ngx_http_v2_connection_t *h2c,
     ngx_http_v2_node_t *node, ngx_uint_t depend, ngx_uint_t exclusive)
 {
-    ngx_queue_t         *children;
-    ngx_http_v2_node_t  *parent, *next;
+    ngx_queue_t         *children, *q;
+    ngx_http_v2_node_t  *parent, *child, *next;
 
     parent = depend ? ngx_http_v2_get_node_by_id(h2c, depend, 0) : NULL;
 
@@ -4003,6 +4004,14 @@ ngx_http_v2_set_dependency(ngx_http_v2_c
     }
 
     if (exclusive) {
+        for (q = ngx_queue_head(children);
+             q != ngx_queue_sentinel(children);
+             q = ngx_queue_next(q))
+        {
+            child = ngx_queue_data(q, ngx_http_v2_node_t, queue);
+            child->parent = node;
+        }
+
         ngx_queue_add(&node->children, children);
         ngx_queue_init(children);
     }


From vbart at nginx.com  Mon Oct 26 14:39:33 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:39:33 +0000
Subject: [nginx] HTTP/2: fix handling of connection errors.
Message-ID: <hg.60f916da7294.1445870373.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/60f916da7294
branches:  
changeset: 6273:60f916da7294
user:      Piotr Sikora <piotrsikora at google.com>
date:      Thu Oct 01 20:25:55 2015 -0700
description:
HTTP/2: fix handling of connection errors.

Previously, nginx worker would crash because of a double free
if client disconnected or timed out before sending all headers.

Found with afl-fuzz.

Signed-off-by: Piotr Sikora <piotrsikora at google.com>

diffstat:

 src/http/v2/ngx_http_v2.c |  12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diffs (29 lines):

diff -r b6a665bf858a -r 60f916da7294 src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Thu Oct 01 20:25:55 2015 -0700
+++ b/src/http/v2/ngx_http_v2.c	Thu Oct 01 20:25:55 2015 -0700
@@ -2377,12 +2377,6 @@ ngx_http_v2_connection_error(ngx_http_v2
         ngx_debug_point();
     }
 
-    if (h2c->state.stream) {
-        h2c->state.stream->out_closed = 1;
-        h2c->state.pool = NULL;
-        ngx_http_v2_close_stream(h2c->state.stream, NGX_HTTP_BAD_REQUEST);
-    }
-
     ngx_http_v2_finalize_connection(h2c, err);
 
     return NULL;
@@ -3814,6 +3808,12 @@ ngx_http_v2_finalize_connection(ngx_http
 
     c = h2c->connection;
 
+    if (h2c->state.stream) {
+        h2c->state.stream->out_closed = 1;
+        h2c->state.pool = NULL;
+        ngx_http_v2_close_stream(h2c->state.stream, NGX_HTTP_BAD_REQUEST);
+    }
+
     h2c->blocked = 1;
 
     if (!c->error && ngx_http_v2_send_goaway(h2c, status) != NGX_ERROR) {


From vbart at nginx.com  Mon Oct 26 14:45:57 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:45:57 +0000
Subject: [nginx] HTTP/2: fixed header block size calculation.
Message-ID: <hg.b2de4a56b860.1445870757.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b2de4a56b860
branches:  
changeset: 6274:b2de4a56b860
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:43:08 2015 +0300
description:
HTTP/2: fixed header block size calculation.

diffstat:

 src/http/v2/ngx_http_v2_filter_module.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff -r 60f916da7294 -r b2de4a56b860 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Thu Oct 01 20:25:55 2015 -0700
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
@@ -222,7 +222,7 @@ ngx_http_v2_header_filter(ngx_http_reque
     }
 
     if (r->headers_out.content_type.len) {
-        len += NGX_HTTP_V2_INT_OCTETS + r->headers_out.content_type.len;
+        len += 1 + NGX_HTTP_V2_INT_OCTETS + r->headers_out.content_type.len;
 
         if (r->headers_out.content_type_len == r->headers_out.content_type.len
             && r->headers_out.charset.len)


From vbart at nginx.com  Mon Oct 26 14:46:00 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:46:00 +0000
Subject: [nginx] HTTP/2: simplified producing of the Last-Modified header.
Message-ID: <hg.39fa0b7e3a67.1445870760.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/39fa0b7e3a67
branches:  
changeset: 6275:39fa0b7e3a67
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:43:08 2015 +0300
description:
HTTP/2: simplified producing of the Last-Modified header.

diffstat:

 src/http/v2/ngx_http_v2_filter_module.c |  5 ++---
 1 files changed, 2 insertions(+), 3 deletions(-)

diffs (15 lines):

diff -r b2de4a56b860 -r 39fa0b7e3a67 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
@@ -484,9 +484,8 @@ ngx_http_v2_header_filter(ngx_http_reque
     {
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LAST_MODIFIED_INDEX);
 
-        p = b->last;
-        b->last = ngx_http_time(b->last + 1, r->headers_out.last_modified_time);
-        *p = (u_char) (b->last - p - 1);
+        *b->last++ = sizeof("Wed, 31 Dec 1986 18:00:00 GMT") - 1;
+        b->last = ngx_http_time(b->last, r->headers_out.last_modified_time);
     }
 
     if (r->headers_out.location && r->headers_out.location->value.len) {


From vbart at nginx.com  Mon Oct 26 14:46:02 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:46:02 +0000
Subject: [nginx] HTTP/2: introduced NGX_HTTP_V2_ENCODE_* macros.
Message-ID: <hg.0efc16d55adb.1445870762.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/0efc16d55adb
branches:  
changeset: 6276:0efc16d55adb
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:43:08 2015 +0300
description:
HTTP/2: introduced NGX_HTTP_V2_ENCODE_* macros.

No functional changes.

diffstat:

 src/http/v2/ngx_http_v2_filter_module.c |  25 ++++++++++++++-----------
 1 files changed, 14 insertions(+), 11 deletions(-)

diffs (104 lines):

diff -r 39fa0b7e3a67 -r 0efc16d55adb src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
@@ -20,6 +20,8 @@
 #define ngx_http_v2_indexed(i)      (128 + (i))
 #define ngx_http_v2_inc_indexed(i)  (64 + (i))
 
+#define NGX_HTTP_V2_ENCODE_RAW            0
+#define NGX_HTTP_V2_ENCODE_HUFF           0x80
 
 #define NGX_HTTP_V2_STATUS_INDEX          8
 #define NGX_HTTP_V2_STATUS_200_INDEX      8
@@ -406,7 +408,7 @@ ngx_http_v2_header_filter(ngx_http_reque
 
     } else {
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_STATUS_INDEX);
-        *b->last++ = 3;
+        *b->last++ = NGX_HTTP_V2_ENCODE_RAW | 3;
         b->last = ngx_sprintf(b->last, "%03ui", r->headers_out.status);
     }
 
@@ -414,11 +416,11 @@ ngx_http_v2_header_filter(ngx_http_reque
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_SERVER_INDEX);
 
         if (clcf->server_tokens) {
-            *b->last++ = sizeof(NGINX_VER) - 1;
+            *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof(NGINX_VER) - 1);
             b->last = ngx_cpymem(b->last, NGINX_VER, sizeof(NGINX_VER) - 1);
 
         } else {
-            *b->last++ = sizeof("nginx") - 1;
+            *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("nginx") - 1);
             b->last = ngx_cpymem(b->last, "nginx", sizeof("nginx") - 1);
         }
     }
@@ -437,7 +439,7 @@ ngx_http_v2_header_filter(ngx_http_reque
         if (r->headers_out.content_type_len == r->headers_out.content_type.len
             && r->headers_out.charset.len)
         {
-            *b->last = 0;
+            *b->last = NGX_HTTP_V2_ENCODE_RAW;
             b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
                                             r->headers_out.content_type.len
                                             + sizeof("; charset=") - 1
@@ -460,7 +462,7 @@ ngx_http_v2_header_filter(ngx_http_reque
             r->headers_out.content_type.data = p;
 
         } else {
-            *b->last = 0;
+            *b->last = NGX_HTTP_V2_ENCODE_RAW;
             b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
                                             r->headers_out.content_type.len);
             b->last = ngx_cpymem(b->last, r->headers_out.content_type.data,
@@ -476,7 +478,7 @@ ngx_http_v2_header_filter(ngx_http_reque
         p = b->last;
         b->last = ngx_sprintf(b->last + 1, "%O",
                               r->headers_out.content_length_n);
-        *p = (u_char) (b->last - p - 1);
+        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (b->last - p - 1);
     }
 
     if (r->headers_out.last_modified == NULL
@@ -484,14 +486,15 @@ ngx_http_v2_header_filter(ngx_http_reque
     {
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LAST_MODIFIED_INDEX);
 
-        *b->last++ = sizeof("Wed, 31 Dec 1986 18:00:00 GMT") - 1;
+        *b->last++ = NGX_HTTP_V2_ENCODE_RAW
+                     | (sizeof("Wed, 31 Dec 1986 18:00:00 GMT") - 1);
         b->last = ngx_http_time(b->last, r->headers_out.last_modified_time);
     }
 
     if (r->headers_out.location && r->headers_out.location->value.len) {
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LOCATION_INDEX);
 
-        *b->last = 0;
+        *b->last = NGX_HTTP_V2_ENCODE_RAW;
         b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
                                         r->headers_out.location->value.len);
         b->last = ngx_cpymem(b->last, r->headers_out.location->value.data,
@@ -501,7 +504,7 @@ ngx_http_v2_header_filter(ngx_http_reque
 #if (NGX_HTTP_GZIP)
     if (r->gzip_vary) {
         *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_VARY_INDEX);
-        *b->last++ = sizeof("Accept-Encoding") - 1;
+        *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("Accept-Encoding") - 1);
         b->last = ngx_cpymem(b->last, "Accept-Encoding",
                              sizeof("Accept-Encoding") - 1);
     }
@@ -558,12 +561,12 @@ ngx_http_v2_header_filter(ngx_http_reque
 
         *p++ = 0;
 
-        *p = 0;
+        *p = NGX_HTTP_V2_ENCODE_RAW;
         p = ngx_http_v2_write_int(p, ngx_http_v2_prefix(7), header[i].key.len);
         ngx_strlow(p, header[i].key.data, header[i].key.len);
         p += header[i].key.len;
 
-        *p = 0;
+        *p = NGX_HTTP_V2_ENCODE_RAW;
         p = ngx_http_v2_write_int(p, ngx_http_v2_prefix(7),
                                   header[i].value.len);
         p = ngx_cpymem(p, header[i].value.data, header[i].value.len);


From vbart at nginx.com  Mon Oct 26 14:46:04 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:46:04 +0000
Subject: [nginx] HTTP/2: fixed splitting of response headers on CONTINUAT...
Message-ID: <hg.b930e598a199.1445870764.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b930e598a199
branches:  
changeset: 6277:b930e598a199
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Sep 28 02:32:44 2015 +0300
description:
HTTP/2: fixed splitting of response headers on CONTINUATION frames.

Previous code has been based on assumption that the header block can only be
splitted at the borders of individual headers.  That wasn't the case and might
result in emitting frames bigger than the frame size limit.

The current approach is to split header blocks by the frame size limit.

diffstat:

 src/http/v2/ngx_http_v2_filter_module.c |  341 +++++++++++++++----------------
 1 files changed, 163 insertions(+), 178 deletions(-)

diffs (truncated from 470 to 300 lines):

diff -r 0efc16d55adb -r b930e598a199 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:43:08 2015 +0300
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Mon Sep 28 02:32:44 2015 +0300
@@ -43,10 +43,8 @@
 
 static u_char *ngx_http_v2_write_int(u_char *pos, ngx_uint_t prefix,
     ngx_uint_t value);
-static void ngx_http_v2_write_headers_head(u_char *pos, size_t length,
-    ngx_uint_t sid, ngx_uint_t end_headers, ngx_uint_t end_stream);
-static void ngx_http_v2_write_continuation_head(u_char *pos, size_t length,
-    ngx_uint_t sid, ngx_uint_t end_headers);
+static ngx_http_v2_out_frame_t *ngx_http_v2_create_headers_frame(
+    ngx_http_request_t *r, u_char *pos, u_char *end);
 
 static ngx_chain_t *ngx_http_v2_send_chain(ngx_connection_t *fc,
     ngx_chain_t *in, off_t limit);
@@ -116,17 +114,14 @@ static ngx_http_output_header_filter_pt 
 static ngx_int_t
 ngx_http_v2_header_filter(ngx_http_request_t *r)
 {
-    u_char                     status, *p, *head;
-    size_t                     len, rest, frame_size;
-    ngx_buf_t                 *b;
+    u_char                     status, *pos, *start, *p;
+    size_t                     len;
     ngx_str_t                  host, location;
-    ngx_uint_t                 i, port, continuation;
-    ngx_chain_t               *cl;
+    ngx_uint_t                 i, port;
     ngx_list_part_t           *part;
     ngx_table_elt_t           *header;
     ngx_connection_t          *fc;
     ngx_http_cleanup_t        *cln;
-    ngx_http_v2_stream_t      *stream;
     ngx_http_v2_out_frame_t   *frame;
     ngx_http_core_loc_conf_t  *clcf;
     ngx_http_core_srv_conf_t  *cscf;
@@ -388,134 +383,117 @@ ngx_http_v2_header_filter(ngx_http_reque
                  + NGX_HTTP_V2_INT_OCTETS + header[i].value.len;
     }
 
-    stream = r->stream;
-    frame_size = stream->connection->frame_size;
-
-    len += NGX_HTTP_V2_FRAME_HEADER_SIZE
-           * ((len + frame_size - 1) / frame_size);
-
-    b = ngx_create_temp_buf(r->pool, len);
-    if (b == NULL) {
+    pos = ngx_palloc(r->pool, len);
+    if (pos == NULL) {
         return NGX_ERROR;
     }
 
-    b->last_buf = r->header_only;
-
-    b->last += NGX_HTTP_V2_FRAME_HEADER_SIZE;
+    start = pos;
 
     if (status) {
-        *b->last++ = status;
+        *pos++ = status;
 
     } else {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_STATUS_INDEX);
-        *b->last++ = NGX_HTTP_V2_ENCODE_RAW | 3;
-        b->last = ngx_sprintf(b->last, "%03ui", r->headers_out.status);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_STATUS_INDEX);
+        *pos++ = NGX_HTTP_V2_ENCODE_RAW | 3;
+        pos = ngx_sprintf(pos, "%03ui", r->headers_out.status);
     }
 
     if (r->headers_out.server == NULL) {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_SERVER_INDEX);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_SERVER_INDEX);
 
         if (clcf->server_tokens) {
-            *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof(NGINX_VER) - 1);
-            b->last = ngx_cpymem(b->last, NGINX_VER, sizeof(NGINX_VER) - 1);
+            *pos++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof(NGINX_VER) - 1);
+            pos = ngx_cpymem(pos, NGINX_VER, sizeof(NGINX_VER) - 1);
 
         } else {
-            *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("nginx") - 1);
-            b->last = ngx_cpymem(b->last, "nginx", sizeof("nginx") - 1);
+            *pos++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("nginx") - 1);
+            pos = ngx_cpymem(pos, "nginx", sizeof("nginx") - 1);
         }
     }
 
     if (r->headers_out.date == NULL) {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_DATE_INDEX);
-        *b->last++ = (u_char) ngx_cached_http_time.len;
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_DATE_INDEX);
+        *pos++ = (u_char) ngx_cached_http_time.len;
 
-        b->last = ngx_cpymem(b->last, ngx_cached_http_time.data,
-                             ngx_cached_http_time.len);
+        pos = ngx_cpymem(pos, ngx_cached_http_time.data,
+                         ngx_cached_http_time.len);
     }
 
     if (r->headers_out.content_type.len) {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_CONTENT_TYPE_INDEX);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_CONTENT_TYPE_INDEX);
 
         if (r->headers_out.content_type_len == r->headers_out.content_type.len
             && r->headers_out.charset.len)
         {
-            *b->last = NGX_HTTP_V2_ENCODE_RAW;
-            b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
-                                            r->headers_out.content_type.len
-                                            + sizeof("; charset=") - 1
-                                            + r->headers_out.charset.len);
+            *pos = NGX_HTTP_V2_ENCODE_RAW;
+            pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(7),
+                                        r->headers_out.content_type.len
+                                        + sizeof("; charset=") - 1
+                                        + r->headers_out.charset.len);
 
-            p = b->last;
+            p = pos;
 
-            b->last = ngx_cpymem(p, r->headers_out.content_type.data,
-                                 r->headers_out.content_type.len);
+            pos = ngx_cpymem(pos, r->headers_out.content_type.data,
+                             r->headers_out.content_type.len);
 
-            b->last = ngx_cpymem(b->last, "; charset=",
-                                 sizeof("; charset=") - 1);
+            pos = ngx_cpymem(pos, "; charset=", sizeof("; charset=") - 1);
 
-            b->last = ngx_cpymem(b->last, r->headers_out.charset.data,
-                                 r->headers_out.charset.len);
+            pos = ngx_cpymem(pos, r->headers_out.charset.data,
+                             r->headers_out.charset.len);
 
             /* update r->headers_out.content_type for possible logging */
 
-            r->headers_out.content_type.len = b->last - p;
+            r->headers_out.content_type.len = pos - p;
             r->headers_out.content_type.data = p;
 
         } else {
-            *b->last = NGX_HTTP_V2_ENCODE_RAW;
-            b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
-                                            r->headers_out.content_type.len);
-            b->last = ngx_cpymem(b->last, r->headers_out.content_type.data,
-                                 r->headers_out.content_type.len);
+            *pos = NGX_HTTP_V2_ENCODE_RAW;
+            pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(7),
+                                        r->headers_out.content_type.len);
+            pos = ngx_cpymem(pos, r->headers_out.content_type.data,
+                             r->headers_out.content_type.len);
         }
     }
 
     if (r->headers_out.content_length == NULL
         && r->headers_out.content_length_n >= 0)
     {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_CONTENT_LENGTH_INDEX);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_CONTENT_LENGTH_INDEX);
 
-        p = b->last;
-        b->last = ngx_sprintf(b->last + 1, "%O",
-                              r->headers_out.content_length_n);
-        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (b->last - p - 1);
+        p = pos;
+        pos = ngx_sprintf(pos + 1, "%O", r->headers_out.content_length_n);
+        *p = NGX_HTTP_V2_ENCODE_RAW | (u_char) (pos - p - 1);
     }
 
     if (r->headers_out.last_modified == NULL
         && r->headers_out.last_modified_time != -1)
     {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LAST_MODIFIED_INDEX);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LAST_MODIFIED_INDEX);
 
-        *b->last++ = NGX_HTTP_V2_ENCODE_RAW
-                     | (sizeof("Wed, 31 Dec 1986 18:00:00 GMT") - 1);
-        b->last = ngx_http_time(b->last, r->headers_out.last_modified_time);
+        *pos++ = NGX_HTTP_V2_ENCODE_RAW
+                 | (sizeof("Wed, 31 Dec 1986 18:00:00 GMT") - 1);
+        pos = ngx_http_time(pos, r->headers_out.last_modified_time);
     }
 
     if (r->headers_out.location && r->headers_out.location->value.len) {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LOCATION_INDEX);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_LOCATION_INDEX);
 
-        *b->last = NGX_HTTP_V2_ENCODE_RAW;
-        b->last = ngx_http_v2_write_int(b->last, ngx_http_v2_prefix(7),
-                                        r->headers_out.location->value.len);
-        b->last = ngx_cpymem(b->last, r->headers_out.location->value.data,
-                                      r->headers_out.location->value.len);
+        *pos = NGX_HTTP_V2_ENCODE_RAW;
+        pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(7),
+                                    r->headers_out.location->value.len);
+        pos = ngx_cpymem(pos, r->headers_out.location->value.data,
+                              r->headers_out.location->value.len);
     }
 
 #if (NGX_HTTP_GZIP)
     if (r->gzip_vary) {
-        *b->last++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_VARY_INDEX);
-        *b->last++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("Accept-Encoding") - 1);
-        b->last = ngx_cpymem(b->last, "Accept-Encoding",
-                             sizeof("Accept-Encoding") - 1);
+        *pos++ = ngx_http_v2_inc_indexed(NGX_HTTP_V2_VARY_INDEX);
+        *pos++ = NGX_HTTP_V2_ENCODE_RAW | (sizeof("Accept-Encoding") - 1);
+        pos = ngx_cpymem(pos, "Accept-Encoding", sizeof("Accept-Encoding") - 1);
     }
 #endif
 
-    continuation = 0;
-    head = b->pos;
-
-    len = b->last - head - NGX_HTTP_V2_FRAME_HEADER_SIZE;
-    rest = frame_size - len;
-
     part = &r->headers_out.headers.part;
     header = part->elts;
 
@@ -535,82 +513,26 @@ ngx_http_v2_header_filter(ngx_http_reque
             continue;
         }
 
-        len = 1 + NGX_HTTP_V2_INT_OCTETS * 2
-              + header[i].key.len
-              + header[i].value.len;
+        *pos++ = 0;
 
-        if (len > rest) {
-            len = b->last - head - NGX_HTTP_V2_FRAME_HEADER_SIZE;
+        *pos = NGX_HTTP_V2_ENCODE_RAW;
+        pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(7),
+                                    header[i].key.len);
+        ngx_strlow(pos, header[i].key.data, header[i].key.len);
+        pos += header[i].key.len;
 
-            if (continuation) {
-                ngx_http_v2_write_continuation_head(head, len,
-                                                    stream->node->id, 0);
-            } else {
-                continuation = 1;
-                ngx_http_v2_write_headers_head(head, len, stream->node->id, 0,
-                                               r->header_only);
-            }
-
-            rest = frame_size;
-            head = b->last;
-
-            b->last += NGX_HTTP_V2_FRAME_HEADER_SIZE;
-        }
-
-        p = b->last;
-
-        *p++ = 0;
-
-        *p = NGX_HTTP_V2_ENCODE_RAW;
-        p = ngx_http_v2_write_int(p, ngx_http_v2_prefix(7), header[i].key.len);
-        ngx_strlow(p, header[i].key.data, header[i].key.len);
-        p += header[i].key.len;
-
-        *p = NGX_HTTP_V2_ENCODE_RAW;
-        p = ngx_http_v2_write_int(p, ngx_http_v2_prefix(7),
-                                  header[i].value.len);
-        p = ngx_cpymem(p, header[i].value.data, header[i].value.len);
-
-        rest -= p - b->last;
-        b->last = p;
+        *pos = NGX_HTTP_V2_ENCODE_RAW;
+        pos = ngx_http_v2_write_int(pos, ngx_http_v2_prefix(7),
+                                    header[i].value.len);
+        pos = ngx_cpymem(pos, header[i].value.data, header[i].value.len);
     }
 
-    len = b->last - head - NGX_HTTP_V2_FRAME_HEADER_SIZE;
-
-    if (continuation) {
-        ngx_http_v2_write_continuation_head(head, len, stream->node->id, 1);
-
-    } else {
-        ngx_http_v2_write_headers_head(head, len, stream->node->id, 1,
-                                       r->header_only);
-    }
-
-    cl = ngx_alloc_chain_link(r->pool);
-    if (cl == NULL) {
-        return NGX_ERROR;
-    }
-
-    cl->buf = b;
-    cl->next = NULL;
-
-    frame = ngx_palloc(r->pool, sizeof(ngx_http_v2_out_frame_t));
+    frame = ngx_http_v2_create_headers_frame(r, start, pos);
     if (frame == NULL) {
         return NGX_ERROR;
     }


From vbart at nginx.com  Mon Oct 26 14:47:25 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:47:25 +0000
Subject: [nginx] HTTP/2: fixed the NGX_HTTP_V2_MAX_FIELD macro.
Message-ID: <hg.b78df0822168.1445870845.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b78df0822168
branches:  
changeset: 6278:b78df0822168
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:46:13 2015 +0300
description:
HTTP/2: fixed the NGX_HTTP_V2_MAX_FIELD macro.

diffstat:

 src/http/v2/ngx_http_v2.h |  3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diffs (13 lines):

diff -r b930e598a199 -r b78df0822168 src/http/v2/ngx_http_v2.h
--- a/src/http/v2/ngx_http_v2.h	Mon Sep 28 02:32:44 2015 +0300
+++ b/src/http/v2/ngx_http_v2.h	Mon Oct 26 17:46:13 2015 +0300
@@ -21,7 +21,8 @@
 #define NGX_HTTP_V2_MAX_FRAME_SIZE       ((1 << 24) - 1)
 
 #define NGX_HTTP_V2_INT_OCTETS           4
-#define NGX_HTTP_V2_MAX_FIELD            ((1 << NGX_HTTP_V2_INT_OCTETS * 7) - 1)
+#define NGX_HTTP_V2_MAX_FIELD                                                 \
+    (127 + (1 << (NGX_HTTP_V2_INT_OCTETS - 1) * 7) - 1)
 
 #define NGX_HTTP_V2_DATA_DISCARD         1
 #define NGX_HTTP_V2_DATA_ERROR           2


From vbart at nginx.com  Mon Oct 26 14:47:27 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:47:27 +0000
Subject: [nginx] HTTP/2: improved the ngx_http_v2_integer_octets(v) macro.
Message-ID: <hg.c72eaf694d99.1445870847.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/c72eaf694d99
branches:  
changeset: 6279:c72eaf694d99
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:46:16 2015 +0300
description:
HTTP/2: improved the ngx_http_v2_integer_octets(v) macro.

Previously, it didn't work well for 0, 127, and 128, returning less than needed.

diffstat:

 src/http/v2/ngx_http_v2_filter_module.c |  7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diffs (17 lines):

diff -r b78df0822168 -r c72eaf694d99 src/http/v2/ngx_http_v2_filter_module.c
--- a/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:46:13 2015 +0300
+++ b/src/http/v2/ngx_http_v2_filter_module.c	Mon Oct 26 17:46:16 2015 +0300
@@ -12,7 +12,12 @@
 #include <ngx_http_v2_module.h>
 
 
-#define ngx_http_v2_integer_octets(v)  (((v) + 127) / 128)
+/*
+ * This returns precise number of octets for values in range 0..253
+ * and estimate number for the rest, but not smaller than required.
+ */
+
+#define ngx_http_v2_integer_octets(v)  (1 + (v) / 127)
 
 #define ngx_http_v2_literal_size(h)                                           \
     (ngx_http_v2_integer_octets(sizeof(h) - 1) + sizeof(h) - 1)


From vbart at nginx.com  Mon Oct 26 14:47:30 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Mon, 26 Oct 2015 14:47:30 +0000
Subject: [nginx] HTTP/2: simplified checking the END_STREAM flag.
Message-ID: <hg.16905ecbb49e.1445870850.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/16905ecbb49e
branches:  
changeset: 6280:16905ecbb49e
user:      Valentin Bartenev <vbart at nginx.com>
date:      Mon Oct 26 17:46:28 2015 +0300
description:
HTTP/2: simplified checking the END_STREAM flag.

No functional changes.

diffstat:

 src/http/v2/ngx_http_v2.c |  8 +++-----
 1 files changed, 3 insertions(+), 5 deletions(-)

diffs (32 lines):

diff -r c72eaf694d99 -r 16905ecbb49e src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Mon Oct 26 17:46:16 2015 +0300
+++ b/src/http/v2/ngx_http_v2.c	Mon Oct 26 17:46:28 2015 +0300
@@ -870,6 +870,8 @@ ngx_http_v2_state_data(ngx_http_v2_conne
         return ngx_http_v2_state_skip_padded(h2c, pos, end);
     }
 
+    stream->in_closed = h2c->state.flags & NGX_HTTP_V2_END_STREAM_FLAG;
+
     h2c->state.stream = stream;
 
     return ngx_http_v2_state_read_data(h2c, pos, end);
@@ -897,8 +899,6 @@ ngx_http_v2_state_read_data(ngx_http_v2_
     }
 
     if (stream->skip_data) {
-        stream->in_closed = h2c->state.flags & NGX_HTTP_V2_END_STREAM_FLAG;
-
         ngx_log_debug1(NGX_LOG_DEBUG_HTTP, h2c->connection->log, 0,
                        "skipping http2 DATA frame, reason: %d",
                        stream->skip_data);
@@ -988,9 +988,7 @@ ngx_http_v2_state_read_data(ngx_http_v2_
                                       ngx_http_v2_state_read_data);
     }
 
-    if (h2c->state.flags & NGX_HTTP_V2_END_STREAM_FLAG) {
-        stream->in_closed = 1;
-
+    if (stream->in_closed) {
         if (r->headers_in.content_length_n < 0) {
             r->headers_in.content_length_n = rb->rest;
 


From mdounin at mdounin.ru  Mon Oct 26 16:44:14 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Mon, 26 Oct 2015 19:44:14 +0300
Subject: [PATCH] HTTP: include SSI header only where it's needed
In-Reply-To: <685e87ae457b29e30ca9.1445729195@piotrsikora.sfo.corp.google.com>
References: <685e87ae457b29e30ca9.1445729195@piotrsikora.sfo.corp.google.com>
Message-ID: <20151026164414.GX48365@mdounin.ru>

Hello!

On Sat, Oct 24, 2015 at 04:26:35PM -0700, Piotr Sikora wrote:

> # HG changeset patch
> # User Piotr Sikora <piotrsikora at google.com>
> # Date 1445649693 25200
> #      Fri Oct 23 18:21:33 2015 -0700
> # Node ID 685e87ae457b29e30ca99e1ae039f4a87ccc1b8e
> # Parent  903e6a14b17e9140ee88f671f1ba2293c512ab3b
> HTTP: include SSI header only where it's needed.

I don't think this change is needed.  Current approach is in line 
with what's done for other headers, and I see no serious enough 
reasons to change things.

-- 
Maxim Dounin
http://nginx.org/


From sorin.v.manole at gmail.com  Mon Oct 26 19:05:54 2015
From: sorin.v.manole at gmail.com (Sorin Manole)
Date: Mon, 26 Oct 2015 21:05:54 +0200
Subject: Processing responses of unbounded sizes from upstream servers
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>

Hey Maxime! The solution you described makes sense and should work. And
yes, you should probably use HTTP_AUX_FILTER_MODULES to register your
handler.
You will have to do a little more juggling with buffers however. From what
I remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY
is to pass on the subrequest response to the client. You probably don't
want that, so you will have to catch the response buffers in the output
handler and not pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c
is probably the best source of inspiration for this.
Anyway, good luck!

2015-10-22 15:30 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com>:

> Hello all,
>
>
> I am currently developing a module that has to send a number of
> subrequests to upstream servers, and aggregate them through application
> logic. I am currently doing that through a post-subrequest handler, using
> the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible
> to receive very large responses from the upstream servers, and I end up
> with the "upstream buffer is too small" error, even after bumping the
> buffer sizes a number of times.
>
> It is my understanding that if I drop this subrequest flag, nginx wouldn't
> try to make the response fit in a single buffer anymore, but then I have no
> idea how to get at that buffer chain - my post-subrequest handler only
> knows about the single buffer in the upstream structure and I haven't been
> able to locate a piece of code that would do things differently.
>
> I suppose it would be possible to use an output filter instead of a
> post-subrequest handler for that use case, would that make sense? And last
> but not least, if I go down that road, can I just move my module
> "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES),
> and still have the rest of module work fine, or will I need to use a second
> module for that?
>
> Thanks a lot in advance!
> Maxime
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151026/c62c7075/attachment.html>

From piotrsikora at google.com  Mon Oct 26 21:45:28 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Mon, 26 Oct 2015 14:45:28 -0700
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL
In-Reply-To: <20151026125825.GU48365@mdounin.ru>
References: <7435401242d6dbfdb4b4.1445729176@piotrsikora.sfo.corp.google.com>
 <20151026125825.GU48365@mdounin.ru>
Message-ID: <CAF-CG+Kz+iUF6TuCBoubcGEKWxVeqxGJcTh3=riKvC+_ns-sjw@mail.gmail.com>

Hey Maxim,

> The NGX_SSL and NGX_OPENSSL macros are distinct as a result of an
> attempt to allow using different SSL libraries by introducing an
> abstraction ngx_ssl_* layer.  I don't think removing this
> distinction is a good idea.

They aren't distinct in the current implementation:
- both are defined at the same time in autotools and there is no way
to get one defined, but not the other,
- source code requires you to have either none or both defined,
otherwise build fails,
- call to abstract ngx_ssl_init() is guarded by NGX_OPENSSL, but it
should be guarded by NGX_SSL,
- OpenSSL is the only ngx_ssl_* implementation.

So, unless there are immediate plans for adding support to other SSL
libraries, I think that they are redundant and NGX_OPENSSL should be
removed.

In any case, at least the NGX_OPENSSL around ngx_ssl_init() should be
replaced with NGX_SSL.

Best regards,
Piotr Sikora


From piotrsikora at google.com  Mon Oct 26 21:45:43 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Mon, 26 Oct 2015 14:45:43 -0700
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL_MD5
In-Reply-To: <20151026125025.GS48365@mdounin.ru>
References: <84b5500257121c6bd3e1.1445729182@piotrsikora.sfo.corp.google.com>
 <20151026125025.GS48365@mdounin.ru>
Message-ID: <CAF-CG+JMgNRRtoXc7G_MX19QDZWerBCwh3zbu2LvJSxajL3BMQ@mail.gmail.com>

Hey Maxim,

> These macros are destinct.  The NGX_HAVE_OPENSSL_MD5_H means that
> we are using "openssl/md5.h" include.  Second one means that we
> are using OpenSSL-style function names, like "MD5_Init", and not
> RSA-like names like "MD5Init".
>
> Such a distinction may be needed when using external MD5 library
> with --with-md5.

Hmm... Are there any useful external MD5 libraries that use
OpenSSL-style function names?

>> diff -r 7435401242d6 -r 84b550025712 auto/lib/md5/conf
>> --- a/auto/lib/md5/conf
>> +++ b/auto/lib/md5/conf
>> @@ -8,8 +8,7 @@ if [ $MD5 != NONE ]; then
>>      if grep MD5_Init $MD5/md5.h 2>&1 >/dev/null; then
>>          # OpenSSL md5
>>          OPENSSL_MD5=YES
>> -        have=NGX_HAVE_OPENSSL_MD5 . auto/have
>> -        have=NGX_OPENSSL_MD5 . auto/have
>> +        have=NGX_HAVE_OPENSSL_MD5_H . auto/have
>
> I don't think this is a typo, rather it tries to define the
> NGX_HAVE_OPENSSL_MD5 macro, currently unused.  Note that it tests
> "md5.h", not "openssl/md5.h".

Right... Shouldn't the unused macro be removed, then?

Best regards,
Piotr Sikora


From piotrsikora at google.com  Mon Oct 26 21:46:33 2015
From: piotrsikora at google.com (Piotr Sikora)
Date: Mon, 26 Oct 2015 14:46:33 -0700
Subject: [PATCH] HTTP: include SSI header only where it's needed
In-Reply-To: <20151026164414.GX48365@mdounin.ru>
References: <685e87ae457b29e30ca9.1445729195@piotrsikora.sfo.corp.google.com>
 <20151026164414.GX48365@mdounin.ru>
Message-ID: <CAF-CG+JahuoEoUypVVewwy7oREsKa9mrVf_+de0tnaRZjYJtrg@mail.gmail.com>

Hey Maxim,

> I don't think this change is needed.  Current approach is in line
> with what's done for other headers, and I see no serious enough
> reasons to change things.

Except that other headers (i.e. cache, ssl, v2) need to be included in
"core" http, because the functions they expose are used in "core"
http, whereas SSI is not. Basically, cache, ssl & v2 are all
compile-time options of the "core" http, whereas SSI is just an http
module and pulling SSI header in "core" http breaks layering &
separation of the modules.

This doesn't really matter if you compile nginx as a big monolith, but
if you want to go DSO route or build modules separately from "core"
http, then "core" http (i.e. ngx_http.h) shouldn't be pulling headers
from non-"core" http modules.

Best regards,
Piotr Sikora


From mdounin at mdounin.ru  Tue Oct 27 01:40:58 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 04:40:58 +0300
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL
In-Reply-To: <CAF-CG+Kz+iUF6TuCBoubcGEKWxVeqxGJcTh3=riKvC+_ns-sjw@mail.gmail.com>
References: <7435401242d6dbfdb4b4.1445729176@piotrsikora.sfo.corp.google.com>
 <20151026125825.GU48365@mdounin.ru>
 <CAF-CG+Kz+iUF6TuCBoubcGEKWxVeqxGJcTh3=riKvC+_ns-sjw@mail.gmail.com>
Message-ID: <20151027014058.GB48365@mdounin.ru>

Hello!

On Mon, Oct 26, 2015 at 02:45:28PM -0700, Piotr Sikora wrote:

> Hey Maxim,
> 
> > The NGX_SSL and NGX_OPENSSL macros are distinct as a result of an
> > attempt to allow using different SSL libraries by introducing an
> > abstraction ngx_ssl_* layer.  I don't think removing this
> > distinction is a good idea.
> 
> They aren't distinct in the current implementation:
> - both are defined at the same time in autotools and there is no way
> to get one defined, but not the other,
> - source code requires you to have either none or both defined,
> otherwise build fails,
> - call to abstract ngx_ssl_init() is guarded by NGX_OPENSSL, but it
> should be guarded by NGX_SSL,
> - OpenSSL is the only ngx_ssl_* implementation.

Both are defined at the same time now, as the only SSL library 
supported is OpenSSL.  They are still distinct on semantic level 
though.

> So, unless there are immediate plans for adding support to other SSL
> libraries, I think that they are redundant and NGX_OPENSSL should be
> removed.

There are no immediate plans to add support for other SSL 
libraries, as well as no plans to remove the bits we 
already have.

> In any case, at least the NGX_OPENSSL around ngx_ssl_init() should be
> replaced with NGX_SSL.

If you want to improve ngx_ssl_* abstraction layer there are lots 
of places to work on.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Tue Oct 27 02:09:27 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 05:09:27 +0300
Subject: [PATCH] Configure: remove redundant NGX_OPENSSL_MD5
In-Reply-To: <CAF-CG+JMgNRRtoXc7G_MX19QDZWerBCwh3zbu2LvJSxajL3BMQ@mail.gmail.com>
References: <84b5500257121c6bd3e1.1445729182@piotrsikora.sfo.corp.google.com>
 <20151026125025.GS48365@mdounin.ru>
 <CAF-CG+JMgNRRtoXc7G_MX19QDZWerBCwh3zbu2LvJSxajL3BMQ@mail.gmail.com>
Message-ID: <20151027020927.GC48365@mdounin.ru>

Hello!

On Mon, Oct 26, 2015 at 02:45:43PM -0700, Piotr Sikora wrote:

> Hey Maxim,
> 
> > These macros are destinct.  The NGX_HAVE_OPENSSL_MD5_H means that
> > we are using "openssl/md5.h" include.  Second one means that we
> > are using OpenSSL-style function names, like "MD5_Init", and not
> > RSA-like names like "MD5Init".
> >
> > Such a distinction may be needed when using external MD5 library
> > with --with-md5.
> 
> Hmm... Are there any useful external MD5 libraries that use
> OpenSSL-style function names?

At least one can be easily found under crypto/md5/ in the OpenSSL 
sources.  Likely recent changes to OpenSSL broke it though. 

(I personally think that --with-md5 and --with-sha1 aren't really 
useful at all, as well as auto/lib/{md5,sha1}.  We have good 
enough internal md5 implementation now (and I have a patch for 
sha1 as well), so there is no real need to use external libraries.  
On the other hand, I don't think that removing --with-md5 and 
--with-sha1 is a good idea either.)


> >> diff -r 7435401242d6 -r 84b550025712 auto/lib/md5/conf
> >> --- a/auto/lib/md5/conf
> >> +++ b/auto/lib/md5/conf
> >> @@ -8,8 +8,7 @@ if [ $MD5 != NONE ]; then
> >>      if grep MD5_Init $MD5/md5.h 2>&1 >/dev/null; then
> >>          # OpenSSL md5
> >>          OPENSSL_MD5=YES
> >> -        have=NGX_HAVE_OPENSSL_MD5 . auto/have
> >> -        have=NGX_OPENSSL_MD5 . auto/have
> >> +        have=NGX_HAVE_OPENSSL_MD5_H . auto/have
> >
> > I don't think this is a typo, rather it tries to define the
> > NGX_HAVE_OPENSSL_MD5 macro, currently unused.  Note that it tests
> > "md5.h", not "openssl/md5.h".
> 
> Right... Shouldn't the unused macro be removed, then?

I don't think it worth the effert, especially considering the fact 
that correct name seems to be NGX_HAVE_OPENSSL_MD5 (broken in 
0.1.9, fixed by introduction of an additional name in 0.1.13).

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Tue Oct 27 02:34:42 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 05:34:42 +0300
Subject: [PATCH] HTTP: include SSI header only where it's needed
In-Reply-To: <CAF-CG+JahuoEoUypVVewwy7oREsKa9mrVf_+de0tnaRZjYJtrg@mail.gmail.com>
References: <685e87ae457b29e30ca9.1445729195@piotrsikora.sfo.corp.google.com>
 <20151026164414.GX48365@mdounin.ru>
 <CAF-CG+JahuoEoUypVVewwy7oREsKa9mrVf_+de0tnaRZjYJtrg@mail.gmail.com>
Message-ID: <20151027023442.GD48365@mdounin.ru>

Hello!

On Mon, Oct 26, 2015 at 02:46:33PM -0700, Piotr Sikora wrote:

> Hey Maxim,
> 
> > I don't think this change is needed.  Current approach is in line
> > with what's done for other headers, and I see no serious enough
> > reasons to change things.
> 
> Except that other headers (i.e. cache, ssl, v2) need to be included in
> "core" http, because the functions they expose are used in "core"
> http, whereas SSI is not. Basically, cache, ssl & v2 are all
> compile-time options of the "core" http, whereas SSI is just an http
> module and pulling SSI header in "core" http breaks layering &
> separation of the modules.

The SSI module is expected to be available for other modules to 
allow them to add their own SSI commands (like the perl module does).  
I don't see how making it available "breaks layering & separation 
of the modules".

The question is only how you'll make it availble - either 
automatically as of now, or via additional #include when needed.

> This doesn't really matter if you compile nginx as a big monolith, but
> if you want to go DSO route or build modules separately from "core"
> http, then "core" http (i.e. ngx_http.h) shouldn't be pulling headers
> from non-"core" http modules.

There are no plans to build all modules separately from core 
http.  Rather, we are working on to allow building _some_ modules 
separately.  And this mostly applies to modules with external 
dependencies (like perl, xslt, geoip) and 3rd party modules.

And as perl depends on SSI, I don't think it will be possible to 
load SSI dynamically without significant changes to SSI's interface.

-- 
Maxim Dounin
http://nginx.org/


From mdounin at mdounin.ru  Tue Oct 27 14:00:39 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 14:00:39 +0000
Subject: [nginx] nginx-1.9.6-RELEASE
Message-ID: <hg.b78018cfaa2f.1445954439.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/b78018cfaa2f
branches:  
changeset: 6281:b78018cfaa2f
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Oct 27 16:47:29 2015 +0300
description:
nginx-1.9.6-RELEASE

diffstat:

 docs/xml/nginx/changes.xml |  84 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 84 insertions(+), 0 deletions(-)

diffs (94 lines):

diff --git a/docs/xml/nginx/changes.xml b/docs/xml/nginx/changes.xml
--- a/docs/xml/nginx/changes.xml
+++ b/docs/xml/nginx/changes.xml
@@ -5,6 +5,90 @@
 <change_log title="nginx">
 
 
+<changes ver="1.9.6" date="27.10.2015">
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/2
+? ??????? ???????? ??? ????????? segmentation fault.<br/>
+??????? Piotr Sikora ? Denis Andzakovic.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process
+when using HTTP/2.<br/>
+Thanks to Piotr Sikora and Denis Andzakovic.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? HTTP/2 ?????????? $server_protocol ???? ??????.
+</para>
+<para lang="en">
+the $server_protocol variable was empty when using HTTP/2.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+SSL-?????????? ? ???????? ? ?????? stream
+????? ?????????? ??????????? ?? ????????.
+</para>
+<para lang="en">
+backend SSL connections in the stream module
+might be timed out unexpectedly.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? ???????? ssl_session_cache
+? ?????? ??????????? ????????
+? ??????? ???????? ??? ????????? segmentation fault.
+</para>
+<para lang="en">
+a segmentation fault might occur in a worker process
+if different ssl_session_cache settings were used
+in different virtual servers.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+nginx/Windows ?? ????????? ? MinGW gcc;
+?????? ????????? ? 1.9.4.<br/>
+??????? Kouhei Sutou.
+</para>
+<para lang="en">
+nginx/Windows could not be built with MinGW gcc;
+the bug had appeared in 1.9.4.<br/>
+Thanks to Kouhei Sutou.
+</para>
+</change>
+
+<change type="bugfix">
+<para lang="ru">
+??? ????????????? ????????? timer_resolution ?? Windows ????? ?? ???????????.
+</para>
+<para lang="en">
+time was not updated when the timer_resolution directive was used on Windows.
+</para>
+</change>
+
+<change>
+<para lang="ru">
+?????????????? ??????????? ? ?????????.<br/>
+??????? Markus Linnala, Kurtis Nusbaum ? Piotr Sikora.
+</para>
+<para lang="en">
+Miscellaneous minor fixes and improvements.<br/>
+Thanks to Markus Linnala, Kurtis Nusbaum and Piotr Sikora.
+</para>
+</change>
+
+</changes>
+
+
 <changes ver="1.9.5" date="22.09.2015">
 
 <change type="feature">

From mdounin at mdounin.ru  Tue Oct 27 14:00:41 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 14:00:41 +0000
Subject: [nginx] release-1.9.6 tag
Message-ID: <hg.f1424eef1966.1445954441.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/f1424eef1966
branches:  
changeset: 6282:f1424eef1966
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Tue Oct 27 16:47:30 2015 +0300
description:
release-1.9.6 tag

diffstat:

 .hgtags |  1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diffs (8 lines):

diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -388,3 +388,4 @@ 3a32d6e7404a79a0973bcd8d0b83181c5bf66074
 e27a215601292872f545a733859e06d01af1017d release-1.9.3
 5cb7e2eed2031e32d2e5422caf9402758c38a6ad release-1.9.4
 942475e10cb47654205ede7ccbe7d568698e665b release-1.9.5
+b78018cfaa2f0ec20494fccb16252daa87c48a31 release-1.9.6


From vl at nginx.com  Tue Oct 27 14:14:40 2015
From: vl at nginx.com (Vladimir Homutov)
Date: Tue, 27 Oct 2015 17:14:40 +0300
Subject: Syslog Unix socket patch
In-Reply-To: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
References: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
Message-ID: <20151027141439.GA29084@vlpc.nginx.com>

On Fri, Oct 23, 2015 at 08:51:56PM +0200, Nils Hermansson wrote:
> # HG changeset patch
> # User Nils Hermansson <3tnica at gmail.com>
> # Date 1445625283 -7200
> #      Fri Oct 23 20:34:43 2015 +0200
> # Node ID 868fc6b3bf69be611118c597578e749c65698b8c
> # Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
> Most standard syslog facilicties do not expect hostname when logging to
> Unix Sockets.
> This patch removes hostname from syslog message when logging to Unix Socket.
> Tested on rsyslog and syslog-ng
>
> diff -r ee16fb0db905 -r 868fc6b3bf69 src/core/ngx_syslog.c
> --- a/src/core/ngx_syslog.c     Tue Oct 20 21:28:38 2015 +0300
> +++ b/src/core/ngx_syslog.c     Fri Oct 23 20:34:43 2015 +0200
> @@ -219,9 +219,17 @@
>      ngx_uint_t  pri;
>
>      pri = peer->facility * 8 + peer->severity;
> -
> -    return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> &ngx_cached_syslog_time,
> +#if (NGX_HAVE_UNIX_DOMAIN)
> +    if  (peer->server.sockaddr->sa_family == AF_UNIX) {
> +        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
> &ngx_cached_syslog_time,
> +                       &peer->tag);
> +    } else {
> +#endif
> +        return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> &ngx_cached_syslog_time,
>                         &ngx_cycle->hostname, &peer->tag);
> +#if (NGX_HAVE_UNIX_DOMAIN)
> +    }
> +#endif
>  }

Hi Nils,

we agree that there is some inconvenience with local syslog and are
considering adding configuration option to select desired behaviour
at this place.

Note that there is a simple solution - to setup your syslog daemon
to expect hostname. See https://trac.nginx.org/nginx/ticket/677
and https://trac.nginx.org/nginx/ticket/783.

Your suggested patch would breaks existing setups and make message
content depend on transport, what is undesirable.


From mhenrion at appnexus.com  Tue Oct 27 16:00:09 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Tue, 27 Oct 2015 16:00:09 +0000
Subject: Processing responses of unbounded sizes from upstream servers
In-Reply-To: <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>,
 <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>

Hey Sorin!

First and foremost, thanks a lot for your answer.

I started working on implementing a body filter for my needs, and am already facing some strange issues. When my body filter is called on the subrequest I sent, despite it being a body filter, I get a buffer chain containing the whole upstream response, including headers. Since this is not a header filter, this has gotten me quite confused... For what it's worth, my module doesn't even register a header filter. Here is some gdb output:


Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
207 {
(gdb) p *in->buf
$1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
  start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
  end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,
  sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}
(gdb)

Unless I'm missing something, this makes it a lot harder for me to actually work on the body of those responses, as I would end up having to parse the HTTP header to find the part I'm interested in.

I'm also at a loss as to the semantics of the last_buf and last_in_chain flags. The module development guide doesn't contain any information about the "last_in_chain" flag; it only talks about last_buf as a marker indicating that the buffer chain is complete. Intuitively, I would think of "last_in_chain" as a marker that we're at the end of the buffer list, but that doesn't make a lot of sense since this would be redundant with the next pointer being NULL. The ngx_buf.h header is equally unhelpful there and I couldn't locate a piece of code that would make the semantics clear yet (some modules seem to consider both flags as being equal, while others aren't).

Last but not least, is there any documentation as to when exactly the output filters are being called? Mine seems to be called two times for the same subrequest. Once with the buffer that I showed above, and a second time with a buffer that is mostly all zero'ed, except for the "sync" and "last_in_chain" flags both set to 1. That seems to be some kind of a flush request, but again, I don't see any documentation on that.

Thanks in advance,
Maxime
________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Sorin Manole [sorin.v.manole at gmail.com]
Sent: Monday, October 26, 2015 8:05 PM
To: nginx-devel at nginx.org
Subject: Re: Processing responses of unbounded sizes from upstream servers

Hey Maxime! The solution you described makes sense and should work. And yes, you should probably use HTTP_AUX_FILTER_MODULES to register your handler.
You will have to do a little more juggling with buffers however. From what I remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY is to pass on the subrequest response to the client. You probably don't want that, so you will have to catch the response buffers in the output handler and not pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c is probably the best source of inspiration for this.
Anyway, good luck!

2015-10-22 15:30 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com<mailto:mhenrion at appnexus.com>>:
Hello all,


I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.

It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the upstream structure and I haven't been able to locate a piece of code that would do things differently.

I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?

Thanks a lot in advance!
Maxime
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org<mailto:nginx-devel at nginx.org>
http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/9c511fb5/attachment.html>

From mhenrion at appnexus.com  Tue Oct 27 16:12:21 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Tue, 27 Oct 2015 16:12:21 +0000
Subject: Processing responses of unbounded sizes from upstream servers
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>, 
 <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>,
 <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A6F599B65@01-exch-nym2.ad.corp.appnexus.com>

I realized that the "pos" field is actually pointing right at the body of that response, so that solves one of my issues.

I would however appreciate any documentation on those numerous flags sometimes set in buffers and their actual meaning.

Maxime
________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Maxime Henrion [mhenrion at appnexus.com]
Sent: Tuesday, October 27, 2015 5:00 PM
To: nginx-devel at nginx.org
Subject: RE: Processing responses of unbounded sizes from upstream servers

Hey Sorin!

First and foremost, thanks a lot for your answer.

I started working on implementing a body filter for my needs, and am already facing some strange issues. When my body filter is called on the subrequest I sent, despite it being a body filter, I get a buffer chain containing the whole upstream response, including headers. Since this is not a header filter, this has gotten me quite confused... For what it's worth, my module doesn't even register a header filter. Here is some gdb output:


Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
207 {
(gdb) p *in->buf
$1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
  start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
  end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,
  sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}
(gdb)

Unless I'm missing something, this makes it a lot harder for me to actually work on the body of those responses, as I would end up having to parse the HTTP header to find the part I'm interested in.

I'm also at a loss as to the semantics of the last_buf and last_in_chain flags. The module development guide doesn't contain any information about the "last_in_chain" flag; it only talks about last_buf as a marker indicating that the buffer chain is complete. Intuitively, I would think of "last_in_chain" as a marker that we're at the end of the buffer list, but that doesn't make a lot of sense since this would be redundant with the next pointer being NULL. The ngx_buf.h header is equally unhelpful there and I couldn't locate a piece of code that would make the semantics clear yet (some modules seem to consider both flags as being equal, while others aren't).

Last but not least, is there any documentation as to when exactly the output filters are being called? Mine seems to be called two times for the same subrequest. Once with the buffer that I showed above, and a second time with a buffer that is mostly all zero'ed, except for the "sync" and "last_in_chain" flags both set to 1. That seems to be some kind of a flush request, but again, I don't see any documentation on that.

Thanks in advance,
Maxime
________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Sorin Manole [sorin.v.manole at gmail.com]
Sent: Monday, October 26, 2015 8:05 PM
To: nginx-devel at nginx.org
Subject: Re: Processing responses of unbounded sizes from upstream servers

Hey Maxime! The solution you described makes sense and should work. And yes, you should probably use HTTP_AUX_FILTER_MODULES to register your handler.
You will have to do a little more juggling with buffers however. From what I remember, the default behaviour when using not NGX_HTTP_SUBREQUEST_IN_MEMORY is to pass on the subrequest response to the client. You probably don't want that, so you will have to catch the response buffers in the output handler and not pass anything to the next filter.
https://trac.nginx.org/nginx/browser/nginx/src/http/modules/ngx_http_ssi_filter_module.c is probably the best source of inspiration for this.
Anyway, good luck!

2015-10-22 15:30 GMT+03:00 Maxime Henrion <mhenrion at appnexus.com<mailto:mhenrion at appnexus.com>>:
Hello all,


I am currently developing a module that has to send a number of subrequests to upstream servers, and aggregate them through application logic. I am currently doing that through a post-subrequest handler, using the NGX_HTTP_SUBREQUEST_IN_MEMORY flag. My problem is that it is possible to receive very large responses from the upstream servers, and I end up with the "upstream buffer is too small" error, even after bumping the buffer sizes a number of times.

It is my understanding that if I drop this subrequest flag, nginx wouldn't try to make the response fit in a single buffer anymore, but then I have no idea how to get at that buffer chain - my post-subrequest handler only knows about the single buffer in the upstream structure and I haven't been able to locate a piece of code that would do things differently.

I suppose it would be possible to use an output filter instead of a post-subrequest handler for that use case, would that make sense? And last but not least, if I go down that road, can I just move my module "declaration" to the HTTP_AUX_FILTER_MODULES variable (from HTTP_MODULES), and still have the rest of module work fine, or will I need to use a second module for that?

Thanks a lot in advance!
Maxime
_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org<mailto:nginx-devel at nginx.org>
http://mailman.nginx.org/mailman/listinfo/nginx-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/4ff91c67/attachment.html>

From mdounin at mdounin.ru  Tue Oct 27 16:25:16 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Tue, 27 Oct 2015 19:25:16 +0300
Subject: Processing responses of unbounded sizes from upstream servers
In-Reply-To: <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>
References: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>
 <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>
 <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>
Message-ID: <20151027162516.GL48365@mdounin.ru>

Hello!

On Tue, Oct 27, 2015 at 04:00:09PM +0000, Maxime Henrion wrote:

> Hey Sorin!
> 
> First and foremost, thanks a lot for your answer.
> 
> I started working on implementing a body filter for my needs, 
> and am already facing some strange issues. When my body filter 
> is called on the subrequest I sent, despite it being a body 
> filter, I get a buffer chain containing the whole upstream 
> response, including headers. Since this is not a header filter, 
> this has gotten me quite confused... For what it's worth, my 
> module doesn't even register a header filter. Here is some gdb 
> output:
> 
> 
> Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
> 207 {
> (gdb) p *in->buf
> $1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
>   start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
>   end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,
>   sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}
> (gdb)
> 
> Unless I'm missing something, this makes it a lot harder for me 
> to actually work on the body of those responses, as I would end 
> up having to parse the HTTP header to find the part I'm 
> interested in.

You are expected to work with data between in->buf->pos and 
in->buf->last (assuming a buffer in memory).

> I'm also at a loss as to the semantics of the last_buf and 
> last_in_chain flags. The module development guide doesn't 
> contain any information about the "last_in_chain" flag; it only 
> talks about last_buf as a marker indicating that the buffer 
> chain is complete. Intuitively, I would think of "last_in_chain" 
> as a marker that we're at the end of the buffer list, but that 
> doesn't make a lot of sense since this would be redundant with 
> the next pointer being NULL. The ngx_buf.h header is equally 
> unhelpful there and I couldn't locate a piece of code that would 
> make the semantics clear yet (some modules seem to consider both 
> flags as being equal, while others aren't).

The last_buf flag marks the last buffer of a request.  The 
last_in_chain flag marks the last buffer of a subrequest.  Looking 
into ngx_http_send_special() may help to understand things.

> Last but not least, is there any documentation as to when 
> exactly the output filters are being called? Mine seems to be 
> called two times for the same subrequest. Once with the buffer 
> that I showed above, and a second time with a buffer that is 
> mostly all zero'ed, except for the "sync" and "last_in_chain" 
> flags both set to 1. That seems to be some kind of a flush 
> request, but again, I don't see any documentation on that.

Use the Source, Luke.

Output filters are called when nginx needs to output something.  
It may happen in various places and is expected to happen at least 
once if a response has body.   Grep for ngx_http_output_filter() 
for more details.

Seriously, trying to code nginx modules without reading it's 
source code won't work.

-- 
Maxim Dounin
http://nginx.org/


From mhenrion at appnexus.com  Tue Oct 27 17:04:07 2015
From: mhenrion at appnexus.com (Maxime Henrion)
Date: Tue, 27 Oct 2015 17:04:07 +0000
Subject: Processing responses of unbounded sizes from upstream servers
In-Reply-To: <20151027162516.GL48365@mdounin.ru>
References: <EB010ACFBA76EF42ADCF5E994F19747A6F5972A5@01-exch-nym2.ad.corp.appnexus.com>
 <CAH19K_uFvHQD1Kg5EgH31BVoUJ6OJp2wR1DrKgHDAJuNkkyePQ@mail.gmail.com>
 <EB010ACFBA76EF42ADCF5E994F19747A6F599B3F@01-exch-nym2.ad.corp.appnexus.com>,
 <20151027162516.GL48365@mdounin.ru>
Message-ID: <EB010ACFBA76EF42ADCF5E994F19747A6F599BD6@01-exch-nym2.ad.corp.appnexus.com>

Hello Maxim,


Thanks for your input, it *is* appreciated.

As someone who has contributed to a large number of open source projects throughout the years, it is however quite annoying to be pointed at the source over and over again.

You seem to be thinking that I've never taken even a quick look at the source code, and that I am sending mails to this list out of pure laziness. As it turns out, I have been reading this codebase on and on since I started this project. I couldn't possibly have gotten as far as I am now without doing that. I have opened and read stuff in the SSI module more times than I can count now, and perused a large number of core source files.

Yes, ultimately, the source code is the one and only source of truth, and yes, the source code undeniably contains the answers to all the questions I have asked so far, in some form or another. However, if you want the luxury to "UTSL" people, you would need a codebase that is properly commented first... I have rarely worked with code that contains as little comments as the nginx codebase. That wouldn't be so bad if there was an API guide or something to begin with!

I'm more than happy to be pointed at code, I really am. And guess what, I hate sending e-mails to development mailing lists to get information probably a lot more than you hate answering them (as it seems you do), and would MUCH rather be able to get at what I need by myself. But sorry to say, the current nginx's codebase does not constitute a proper documentation on its internals for someone who hasn't had prior experience with the codebase.

So yeah, I'm fully aware that trying to develop nginx modules without reading the source code won't work. I'm not doing that though, not even remotely...

Regards,
Maxime

_______________________________________
From: nginx-devel [nginx-devel-bounces at nginx.org] on behalf of Maxim Dounin [mdounin at mdounin.ru]
Sent: Tuesday, October 27, 2015 5:25 PM
To: nginx-devel at nginx.org
Subject: Re: Processing responses of unbounded sizes from upstream servers

Hello!

On Tue, Oct 27, 2015 at 04:00:09PM +0000, Maxime Henrion wrote:

> Hey Sorin!
>
> First and foremost, thanks a lot for your answer.
>
> I started working on implementing a body filter for my needs,
> and am already facing some strange issues. When my body filter
> is called on the subrequest I sent, despite it being a body
> filter, I get a buffer chain containing the whole upstream
> response, including headers. Since this is not a header filter,
> this has gotten me quite confused... For what it's worth, my
> module doesn't even register a header filter. Here is some gdb
> output:
>
>
> Breakpoint 1, ngx_http_dispatcher_body_filter (r=0x5402050, in=0x5403b38) at /home/mhenrion/dispatcher/ngx_http_dispatcher_module.c:207
> 207 {
> (gdb) p *in->buf
> $1 = {pos = 0x5404441 "N", last = 0x5404498 "", file_pos = 0, file_last = 0,
>   start = 0x54043c0 "HTTP/1.1 200 OK\r\n[more data here that I stripped]",
>   end = 0x54083c0 "", tag = 0x7671e0 <ngx_http_proxy_module>, file = 0x0, shadow = 0x5403698, temporary = 1, memory = 0, mmap = 0, recycled = 0, in_file = 0, flush = 0,
>   sync = 0, last_buf = 0, last_in_chain = 0, last_shadow = 1, temp_file = 0, num = 0}
> (gdb)
>
> Unless I'm missing something, this makes it a lot harder for me
> to actually work on the body of those responses, as I would end
> up having to parse the HTTP header to find the part I'm
> interested in.

You are expected to work with data between in->buf->pos and
in->buf->last (assuming a buffer in memory).

> I'm also at a loss as to the semantics of the last_buf and
> last_in_chain flags. The module development guide doesn't
> contain any information about the "last_in_chain" flag; it only
> talks about last_buf as a marker indicating that the buffer
> chain is complete. Intuitively, I would think of "last_in_chain"
> as a marker that we're at the end of the buffer list, but that
> doesn't make a lot of sense since this would be redundant with
> the next pointer being NULL. The ngx_buf.h header is equally
> unhelpful there and I couldn't locate a piece of code that would
> make the semantics clear yet (some modules seem to consider both
> flags as being equal, while others aren't).

The last_buf flag marks the last buffer of a request.  The
last_in_chain flag marks the last buffer of a subrequest.  Looking
into ngx_http_send_special() may help to understand things.

> Last but not least, is there any documentation as to when
> exactly the output filters are being called? Mine seems to be
> called two times for the same subrequest. Once with the buffer
> that I showed above, and a second time with a buffer that is
> mostly all zero'ed, except for the "sync" and "last_in_chain"
> flags both set to 1. That seems to be some kind of a flush
> request, but again, I don't see any documentation on that.

Use the Source, Luke.

Output filters are called when nginx needs to output something.
It may happen in various places and is expected to happen at least
once if a response has body.   Grep for ngx_http_output_filter()
for more details.

Seriously, trying to code nginx modules without reading it's
source code won't work.

--
Maxim Dounin
http://nginx.org/

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel


From 3tnica at gmail.com  Tue Oct 27 18:27:04 2015
From: 3tnica at gmail.com (Nils Hermansson)
Date: Tue, 27 Oct 2015 19:27:04 +0100
Subject: Syslog Unix socket patch
In-Reply-To: <20151027141439.GA29084@vlpc.nginx.com>
References: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
 <20151027141439.GA29084@vlpc.nginx.com>
Message-ID: <CAJp36wXyP=EZbrkWZvQNTdc0iNXHGenymMj0fooY_rFFD-kTFQ@mail.gmail.com>

Hello,
I could write a patch for that case. What would the recommended config
format be?

Example:
hostname=(on/off)  default is on

access_log syslog:server=unix:/var/log/nginx.sock,hostname=off;


Regards


On Tue, Oct 27, 2015 at 3:14 PM, Vladimir Homutov <vl at nginx.com> wrote:

> On Fri, Oct 23, 2015 at 08:51:56PM +0200, Nils Hermansson wrote:
> > # HG changeset patch
> > # User Nils Hermansson <3tnica at gmail.com>
> > # Date 1445625283 -7200
> > #      Fri Oct 23 20:34:43 2015 +0200
> > # Node ID 868fc6b3bf69be611118c597578e749c65698b8c
> > # Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
> > Most standard syslog facilicties do not expect hostname when logging to
> > Unix Sockets.
> > This patch removes hostname from syslog message when logging to Unix
> Socket.
> > Tested on rsyslog and syslog-ng
> >
> > diff -r ee16fb0db905 -r 868fc6b3bf69 src/core/ngx_syslog.c
> > --- a/src/core/ngx_syslog.c     Tue Oct 20 21:28:38 2015 +0300
> > +++ b/src/core/ngx_syslog.c     Fri Oct 23 20:34:43 2015 +0200
> > @@ -219,9 +219,17 @@
> >      ngx_uint_t  pri;
> >
> >      pri = peer->facility * 8 + peer->severity;
> > -
> > -    return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> > &ngx_cached_syslog_time,
> > +#if (NGX_HAVE_UNIX_DOMAIN)
> > +    if  (peer->server.sockaddr->sa_family == AF_UNIX) {
> > +        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
> > &ngx_cached_syslog_time,
> > +                       &peer->tag);
> > +    } else {
> > +#endif
> > +        return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> > &ngx_cached_syslog_time,
> >                         &ngx_cycle->hostname, &peer->tag);
> > +#if (NGX_HAVE_UNIX_DOMAIN)
> > +    }
> > +#endif
> >  }
>
> Hi Nils,
>
> we agree that there is some inconvenience with local syslog and are
> considering adding configuration option to select desired behaviour
> at this place.
>
> Note that there is a simple solution - to setup your syslog daemon
> to expect hostname. See https://trac.nginx.org/nginx/ticket/677
> and https://trac.nginx.org/nginx/ticket/783.
>
> Your suggested patch would breaks existing setups and make message
> content depend on transport, what is undesirable.
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/f2dd3dde/attachment.html>

From 3tnica at gmail.com  Tue Oct 27 19:21:46 2015
From: 3tnica at gmail.com (Nils Hermansson)
Date: Tue, 27 Oct 2015 20:21:46 +0100
Subject: Syslog Unix socket patch
In-Reply-To: <20151027141439.GA29084@vlpc.nginx.com>
References: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
 <20151027141439.GA29084@vlpc.nginx.com>
Message-ID: <CAJp36wV4G66G-foEehiw+PtP7=UBsurmHX+Ev1qLJYtewyqrtQ@mail.gmail.com>

Created a new patch as disabling hostame as a configuration option.

# HG changeset patch
# User Nils Hermansson <3tnica at gmail.com>
# Date 1445973553 -3600
#      Tue Oct 27 20:19:13 2015 +0100
# Node ID 7e4f788192183a3053cd9478ee136bad269a2375
# Parent  f1424eef1966c966e5403c44e3d9470e0af5b7e9
Syslog configuration for disabling sending of hostname.
hostname=(on/off) default on

diff -r f1424eef1966 -r 7e4f78819218 src/core/ngx_syslog.c
--- a/src/core/ngx_syslog.c     Tue Oct 27 16:47:30 2015 +0300
+++ b/src/core/ngx_syslog.c     Tue Oct 27 20:19:13 2015 +0100
@@ -42,6 +42,7 @@
     peer->pool = cf->pool;
     peer->facility = NGX_CONF_UNSET_UINT;
     peer->severity = NGX_CONF_UNSET_UINT;
+    peer->hostname = NGX_CONF_UNSET_UINT;

     if (ngx_syslog_parse_args(cf, peer) != NGX_CONF_OK) {
         return NGX_CONF_ERROR;
@@ -53,6 +54,10 @@
         return NGX_CONF_ERROR;
     }

+    if (peer->hostname == NGX_CONF_UNSET_UINT) {
+        peer->hostname = 1; /* enabled */
+    }
+
     if (peer->facility == NGX_CONF_UNSET_UINT) {
         peer->facility = 23; /* local7 */
     }
@@ -193,6 +198,12 @@

             peer->tag.data = p + 4;
             peer->tag.len = len - 4;
+        } else if (ngx_strncmp(p, "hostname=", 9) == 0) {
+               if (ngx_strcmp(p + 9, "off") == 0) {
+                    peer->hostname = 0;
+               } else {
+                    peer->hostname = 1;
+                }

         } else {
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
@@ -220,6 +231,11 @@

     pri = peer->facility * 8 + peer->severity;

+    if (peer->hostname == 0) {
+        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
&ngx_cached_syslog_time,
+                           &peer->tag);
+    }
+
     return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
&ngx_cached_syslog_time,
                        &ngx_cycle->hostname, &peer->tag);
 }
diff -r f1424eef1966 -r 7e4f78819218 src/core/ngx_syslog.h
--- a/src/core/ngx_syslog.h     Tue Oct 27 16:47:30 2015 +0300
+++ b/src/core/ngx_syslog.h     Tue Oct 27 20:19:13 2015 +0100
@@ -12,6 +12,7 @@
     ngx_pool_t       *pool;
     ngx_uint_t        facility;
     ngx_uint_t        severity;
+    ngx_uint_t        hostname;
     ngx_str_t         tag;

     ngx_addr_t        server;

On Tue, Oct 27, 2015 at 3:14 PM, Vladimir Homutov <vl at nginx.com> wrote:

> On Fri, Oct 23, 2015 at 08:51:56PM +0200, Nils Hermansson wrote:
> > # HG changeset patch
> > # User Nils Hermansson <3tnica at gmail.com>
> > # Date 1445625283 -7200
> > #      Fri Oct 23 20:34:43 2015 +0200
> > # Node ID 868fc6b3bf69be611118c597578e749c65698b8c
> > # Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
> > Most standard syslog facilicties do not expect hostname when logging to
> > Unix Sockets.
> > This patch removes hostname from syslog message when logging to Unix
> Socket.
> > Tested on rsyslog and syslog-ng
> >
> > diff -r ee16fb0db905 -r 868fc6b3bf69 src/core/ngx_syslog.c
> > --- a/src/core/ngx_syslog.c     Tue Oct 20 21:28:38 2015 +0300
> > +++ b/src/core/ngx_syslog.c     Fri Oct 23 20:34:43 2015 +0200
> > @@ -219,9 +219,17 @@
> >      ngx_uint_t  pri;
> >
> >      pri = peer->facility * 8 + peer->severity;
> > -
> > -    return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> > &ngx_cached_syslog_time,
> > +#if (NGX_HAVE_UNIX_DOMAIN)
> > +    if  (peer->server.sockaddr->sa_family == AF_UNIX) {
> > +        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
> > &ngx_cached_syslog_time,
> > +                       &peer->tag);
> > +    } else {
> > +#endif
> > +        return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> > &ngx_cached_syslog_time,
> >                         &ngx_cycle->hostname, &peer->tag);
> > +#if (NGX_HAVE_UNIX_DOMAIN)
> > +    }
> > +#endif
> >  }
>
> Hi Nils,
>
> we agree that there is some inconvenience with local syslog and are
> considering adding configuration option to select desired behaviour
> at this place.
>
> Note that there is a simple solution - to setup your syslog daemon
> to expect hostname. See https://trac.nginx.org/nginx/ticket/677
> and https://trac.nginx.org/nginx/ticket/783.
>
> Your suggested patch would breaks existing setups and make message
> content depend on transport, what is undesirable.
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/fd80fa5f/attachment.html>

From vbart at nginx.com  Tue Oct 27 20:17:31 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Tue, 27 Oct 2015 20:17:31 +0000
Subject: [nginx] Version bump.
Message-ID: <hg.86b5f146b121.1445977051.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/86b5f146b121
branches:  
changeset: 6283:86b5f146b121
user:      Valentin Bartenev <vbart at nginx.com>
date:      Tue Oct 27 23:16:29 2015 +0300
description:
Version bump.

diffstat:

 src/core/nginx.h |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (14 lines):

diff -r f1424eef1966 -r 86b5f146b121 src/core/nginx.h
--- a/src/core/nginx.h	Tue Oct 27 16:47:30 2015 +0300
+++ b/src/core/nginx.h	Tue Oct 27 23:16:29 2015 +0300
@@ -9,8 +9,8 @@
 #define _NGINX_H_INCLUDED_
 
 
-#define nginx_version      1009006
-#define NGINX_VERSION      "1.9.6"
+#define nginx_version      1009007
+#define NGINX_VERSION      "1.9.7"
 #define NGINX_VER          "nginx/" NGINX_VERSION
 
 #ifdef NGX_BUILD


From vbart at nginx.com  Tue Oct 27 20:17:34 2015
From: vbart at nginx.com (Valentin Bartenev)
Date: Tue, 27 Oct 2015 20:17:34 +0000
Subject: [nginx] HTTP/2: fixed spelling.
Message-ID: <hg.66ee1c5cb6aa.1445977054.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/66ee1c5cb6aa
branches:  
changeset: 6284:66ee1c5cb6aa
user:      Valentin Bartenev <vbart at nginx.com>
date:      Tue Oct 27 23:16:35 2015 +0300
description:
HTTP/2: fixed spelling.

diffstat:

 src/http/v2/ngx_http_v2.c |  4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diffs (21 lines):

diff -r 86b5f146b121 -r 66ee1c5cb6aa src/http/v2/ngx_http_v2.c
--- a/src/http/v2/ngx_http_v2.c	Tue Oct 27 23:16:29 2015 +0300
+++ b/src/http/v2/ngx_http_v2.c	Tue Oct 27 23:16:35 2015 +0300
@@ -1664,7 +1664,7 @@ ngx_http_v2_state_process_header(ngx_htt
     h->key.len = header->name.len;
     h->key.data = header->name.data;
 
-    /* TODO Optimization: precalculate hash and hadnler for indexed headers. */
+    /* TODO Optimization: precalculate hash and handler for indexed headers. */
     h->hash = ngx_hash_key(h->key.data, h->key.len);
 
     h->value.len = header->value.len;
@@ -1818,7 +1818,7 @@ ngx_http_v2_state_priority(ngx_http_v2_c
     if (depend == h2c->state.sid) {
         ngx_log_error(NGX_LOG_INFO, h2c->connection->log, 0,
                       "client sent PRIORITY frame for stream %ui "
-                      "with incorrect dependancy", h2c->state.sid);
+                      "with incorrect dependency", h2c->state.sid);
 
         node = ngx_http_v2_get_node_by_id(h2c, h2c->state.sid, 0);
 


From joel.cunningham at me.com  Tue Oct 27 20:52:43 2015
From: joel.cunningham at me.com (Joel Cunningham)
Date: Tue, 27 Oct 2015 20:52:43 +0000 (GMT)
Subject: ngx_array.c allocation question
Message-ID: <cbe927e6-a70b-4135-b2c6-ce7c72950e86@me.com>

An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/d50c50f0/attachment.html>

From vbart at nginx.com  Tue Oct 27 21:42:11 2015
From: vbart at nginx.com (Valentin V. Bartenev)
Date: Wed, 28 Oct 2015 00:42:11 +0300
Subject: ngx_array.c allocation question
In-Reply-To: <cbe927e6-a70b-4135-b2c6-ce7c72950e86@me.com>
References: <cbe927e6-a70b-4135-b2c6-ce7c72950e86@me.com>
Message-ID: <2505775.0o7cBn1xFN@vbart-laptop>

On Tuesday 27 October 2015 20:52:43 Joel Cunningham wrote:
> Hi,
> 
> I'm fairly new to the NGINX sources and have been studying ngx_array.c (from 
1.9.5), particularly how the array grows when full.
> 
> I'm having some confusion about how pool memory isn't leaked when the new 
array size can not fit within the current pool allocation and we call 
ngx_palloc().  The new allocation is assigned to the local variable new, which 
is eventually assigned to a->elts, but the old memory that was claimed out of 
the previous pool allocation (and tracked by a->elts) isn't released/marked as 
no longer used in the pool.  I would have expected some manipulation of p-
>d.last, similarly to how ngx_array_destroy() releases the memory from the 
pool allocation.  Is there something I'm not understanding about the way array 
uses the memory pool?
> 
> It seems like over time an array that continues to grow (past the current 
pool allocation) would leave memory claimed within the pool allocation that 
it's no longer using or tracking
> 

It's ok, since pools usually have relatively short lives.

  wbr, Valentin V. Bartenev


From joel.cunningham at me.com  Tue Oct 27 22:51:53 2015
From: joel.cunningham at me.com (Joel Cunningham)
Date: Tue, 27 Oct 2015 22:51:53 +0000 (GMT)
Subject: ngx_array.c allocation question
Message-ID: <6c23474a-8a34-47db-a528-503c9c8875ee@me.com>

An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151027/09fd34cd/attachment.html>

From vbart at nginx.com  Wed Oct 28 13:43:28 2015
From: vbart at nginx.com (Valentin V. Bartenev)
Date: Wed, 28 Oct 2015 16:43:28 +0300
Subject: ngx_array.c allocation question
In-Reply-To: <6c23474a-8a34-47db-a528-503c9c8875ee@me.com>
References: <6c23474a-8a34-47db-a528-503c9c8875ee@me.com>
Message-ID: <3778660.BTuupPRgll@vbart-workstation>

On Tuesday 27 October 2015 22:51:53 Joel Cunningham wrote:
> Request pools seem short lived, but the cycle pool sticks around for a long time.
> 
> I'm working on a custom module that has state which should persist the entire uptime of NGINX.  I was going to use an array allocated out of the cycle pool, but now I'm not sure that's a good idea.  My array could grow and potentially leak memory in the cycle pool.  Maybe the cycle pool allocation is big enough we never actually exceed the allocation and each array grow will continue to use the current allocation?  Default size is 16KB.  My module could have its own pool, but the same leak could happen within that.
> 
> Would you say this was an intentional design decision or a bug?  Doesn't look like it would be hard to fix assuming my understanding of ngx_array.c is correct.
> 
[..]

It's an intentional design decision.  Moreover, there's no easy way
to return memory to pool.  Even ngx_array_destroy() returns memory
only in very specific case, when the destruction is happened right
after the allocation, so it works only for some temporary arrays.

  wbr, Valentin V. Bartenev


From levb at yahoo.com  Wed Oct 28 15:24:39 2015
From: levb at yahoo.com (Lev Brouk)
Date: Wed, 28 Oct 2015 15:24:39 +0000 (UTC)
Subject: ngx_array.c allocation question
In-Reply-To: <3778660.BTuupPRgll@vbart-workstation>
References: <3778660.BTuupPRgll@vbart-workstation>
Message-ID: <841484947.933808.1446045879964.JavaMail.yahoo@mail.yahoo.com>

Valentin and Joel,
It would appear that a part of Joel's (and my own) concern could be alleviated if ngx_array_push called ngx_pfree on the old memory block after it grew it in?https://github.com/nginx/nginx/blob/master/src/core/ngx_array.c#L76. I understand that (a) doing this would still leak all the "small" allocs, and (b) ngx_pfree call is potentially slow since it walks the chain of large allocations in the pool. But that could be an option, perhaps?
That aside, I too am wondering what memory management options (other than malloc) are available for data structures with long lifespans, or if there are any relevant development plans. It is certainly appealing to be able to use ngx_array_t and ngx_list_t in these "long-lived" use-cases.
- Lev 


     On Wednesday, October 28, 2015 6:43 AM, Valentin V. Bartenev <vbart at nginx.com> wrote:
   

 On Tuesday 27 October 2015 22:51:53 Joel Cunningham wrote:
> Request pools seem short lived, but the cycle pool sticks around for a long time.
> 
> I'm working on a custom module that has state which should persist the entire uptime of NGINX.? I was going to use an array allocated out of the cycle pool, but now I'm not sure that's a good idea.? My array could grow and potentially leak memory in the cycle pool.? Maybe the cycle pool allocation is big enough we never actually exceed the allocation and each array grow will continue to use the current allocation?? Default size is 16KB.? My module could have its own pool, but the same leak could happen within that.
> 
> Would you say this was an intentional design decision or a bug?? Doesn't look like it would be hard to fix assuming my understanding of ngx_array.c is correct.
> 
[..]

It's an intentional design decision.? Moreover, there's no easy way
to return memory to pool.? Even ngx_array_destroy() returns memory
only in very specific case, when the destruction is happened right
after the allocation, so it works only for some temporary arrays.

? wbr, Valentin V. Bartenev

_______________________________________________
nginx-devel mailing list
nginx-devel at nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel


  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151028/9fce6061/attachment.html>

From vbart at nginx.com  Wed Oct 28 15:49:08 2015
From: vbart at nginx.com (Valentin V. Bartenev)
Date: Wed, 28 Oct 2015 18:49:08 +0300
Subject: ngx_array.c allocation question
In-Reply-To: <841484947.933808.1446045879964.JavaMail.yahoo@mail.yahoo.com>
References: <3778660.BTuupPRgll@vbart-workstation>
 <841484947.933808.1446045879964.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <5186065.W9mrbW9x6F@vbart-workstation>

On Wednesday 28 October 2015 15:24:39 Lev Brouk wrote:
> Valentin and Joel,
> It would appear that a part of Joel's (and my own) concern could be alleviated if ngx_array_push called ngx_pfree on the old memory block after it grew it in https://github.com/nginx/nginx/blob/master/src/core/ngx_array.c#L76. I understand that (a) doing this would still leak all the "small" allocs, and (b) ngx_pfree call is potentially slow since it walks the chain of large allocations in the pool. But that could be an option, perhaps?

We usually don't have so big arrays in nginx that ngx_pfree can free.
So, calling it will be waste of CPU cycles most of the time.

> That aside, I too am wondering what memory management options (other than malloc) are available for data structures with long lifespans, or if there are any relevant development plans. It is certainly appealing to be able to use ngx_array_t and ngx_list_t in these "long-lived" use-cases.

AFAIK, no any dev plans at the moment.  I'd suggest just to implement
some simple reuse mechanism for your data structures.  That's what
nginx does for its chains and buffers that are reused during the
processing of request.

  wbr, Valentin V. Bartenev


From joel.cunningham at me.com  Wed Oct 28 16:23:10 2015
From: joel.cunningham at me.com (Joel Cunningham)
Date: Wed, 28 Oct 2015 16:23:10 +0000 (GMT)
Subject: ngx_array.c allocation question
Message-ID: <7357678a-deb0-484a-94bf-547dc0a5c5e4@me.com>

An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151028/5e29d0a4/attachment.html>

From vl at nginx.com  Thu Oct 29 09:16:52 2015
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 29 Oct 2015 09:16:52 +0000
Subject: [nginx] Syslog: added "nohostname" option.
Message-ID: <hg.a6a2016b8e31.1446110212.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/a6a2016b8e31
branches:  
changeset: 6286:a6a2016b8e31
user:      Vladimir Homutov <vl at nginx.com>
date:      Mon Oct 26 19:06:42 2015 +0300
description:
Syslog: added "nohostname" option.

The option disables sending hostname in the syslog message header.  This is
useful with syslog daemons that do not expect it (tickets #677 and #783).

diffstat:

 src/core/ngx_syslog.c |  8 ++++++++
 src/core/ngx_syslog.h |  3 ++-
 2 files changed, 10 insertions(+), 1 deletions(-)

diffs (38 lines):

diff -r 1f26bf65b1bc -r a6a2016b8e31 src/core/ngx_syslog.c
--- a/src/core/ngx_syslog.c	Tue Oct 27 23:16:35 2015 +0300
+++ b/src/core/ngx_syslog.c	Mon Oct 26 19:06:42 2015 +0300
@@ -194,6 +194,9 @@ ngx_syslog_parse_args(ngx_conf_t *cf, ng
             peer->tag.data = p + 4;
             peer->tag.len = len - 4;
 
+        } else if (len == 10 && ngx_strncmp(p, "nohostname", 10) == 0) {
+            peer->nohostname = 1;
+
         } else {
             ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                                "unknown syslog parameter \"%s\"", p);
@@ -220,6 +223,11 @@ ngx_syslog_add_header(ngx_syslog_peer_t 
 
     pri = peer->facility * 8 + peer->severity;
 
+    if (peer->nohostname) {
+        return ngx_sprintf(buf, "<%ui>%V %V: ", pri, &ngx_cached_syslog_time,
+                           &peer->tag);
+    }
+
     return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri, &ngx_cached_syslog_time,
                        &ngx_cycle->hostname, &peer->tag);
 }
diff -r 1f26bf65b1bc -r a6a2016b8e31 src/core/ngx_syslog.h
--- a/src/core/ngx_syslog.h	Tue Oct 27 23:16:35 2015 +0300
+++ b/src/core/ngx_syslog.h	Mon Oct 26 19:06:42 2015 +0300
@@ -16,7 +16,8 @@ typedef struct {
 
     ngx_addr_t        server;
     ngx_connection_t  conn;
-    ngx_uint_t        busy;  /* unsigned busy:1; */
+    unsigned          busy:1;
+    unsigned          nohostname:1;
 } ngx_syslog_peer_t;
 
 


From mat999 at gmail.com  Thu Oct 29 10:02:35 2015
From: mat999 at gmail.com (SplitIce)
Date: Thu, 29 Oct 2015 21:02:35 +1100
Subject: Syslog Unix socket patch
In-Reply-To: <CAJp36wV4G66G-foEehiw+PtP7=UBsurmHX+Ev1qLJYtewyqrtQ@mail.gmail.com>
References: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
 <20151027141439.GA29084@vlpc.nginx.com>
 <CAJp36wV4G66G-foEehiw+PtP7=UBsurmHX+Ev1qLJYtewyqrtQ@mail.gmail.com>
Message-ID: <CALFfGYavKmW-5On9fVo2hvKMY2M=RWF0rXiq1pPjG6T4H5v3FQ@mail.gmail.com>

In case its relevant we have been running a very similar patch Nils since
the introduction of the syslog feature.

Since the default and expected format on the /dev/log socket is to write
without hostname perhaps unix socket should default to hostname=0 ?

On Wed, Oct 28, 2015 at 6:21 AM, Nils Hermansson <3tnica at gmail.com> wrote:

> Created a new patch as disabling hostame as a configuration option.
>
> # HG changeset patch
> # User Nils Hermansson <3tnica at gmail.com>
> # Date 1445973553 -3600
> #      Tue Oct 27 20:19:13 2015 +0100
> # Node ID 7e4f788192183a3053cd9478ee136bad269a2375
> # Parent  f1424eef1966c966e5403c44e3d9470e0af5b7e9
> Syslog configuration for disabling sending of hostname.
> hostname=(on/off) default on
>
> diff -r f1424eef1966 -r 7e4f78819218 src/core/ngx_syslog.c
> --- a/src/core/ngx_syslog.c     Tue Oct 27 16:47:30 2015 +0300
> +++ b/src/core/ngx_syslog.c     Tue Oct 27 20:19:13 2015 +0100
> @@ -42,6 +42,7 @@
>      peer->pool = cf->pool;
>      peer->facility = NGX_CONF_UNSET_UINT;
>      peer->severity = NGX_CONF_UNSET_UINT;
> +    peer->hostname = NGX_CONF_UNSET_UINT;
>
>      if (ngx_syslog_parse_args(cf, peer) != NGX_CONF_OK) {
>          return NGX_CONF_ERROR;
> @@ -53,6 +54,10 @@
>          return NGX_CONF_ERROR;
>      }
>
> +    if (peer->hostname == NGX_CONF_UNSET_UINT) {
> +        peer->hostname = 1; /* enabled */
> +    }
> +
>      if (peer->facility == NGX_CONF_UNSET_UINT) {
>          peer->facility = 23; /* local7 */
>      }
> @@ -193,6 +198,12 @@
>
>              peer->tag.data = p + 4;
>              peer->tag.len = len - 4;
> +        } else if (ngx_strncmp(p, "hostname=", 9) == 0) {
> +               if (ngx_strcmp(p + 9, "off") == 0) {
> +                    peer->hostname = 0;
> +               } else {
> +                    peer->hostname = 1;
> +                }
>
>          } else {
>              ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
> @@ -220,6 +231,11 @@
>
>      pri = peer->facility * 8 + peer->severity;
>
> +    if (peer->hostname == 0) {
> +        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
> &ngx_cached_syslog_time,
> +                           &peer->tag);
> +    }
> +
>      return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
> &ngx_cached_syslog_time,
>                         &ngx_cycle->hostname, &peer->tag);
>  }
> diff -r f1424eef1966 -r 7e4f78819218 src/core/ngx_syslog.h
> --- a/src/core/ngx_syslog.h     Tue Oct 27 16:47:30 2015 +0300
> +++ b/src/core/ngx_syslog.h     Tue Oct 27 20:19:13 2015 +0100
> @@ -12,6 +12,7 @@
>      ngx_pool_t       *pool;
>      ngx_uint_t        facility;
>      ngx_uint_t        severity;
> +    ngx_uint_t        hostname;
>      ngx_str_t         tag;
>
>      ngx_addr_t        server;
>
> On Tue, Oct 27, 2015 at 3:14 PM, Vladimir Homutov <vl at nginx.com> wrote:
>
>> On Fri, Oct 23, 2015 at 08:51:56PM +0200, Nils Hermansson wrote:
>> > # HG changeset patch
>> > # User Nils Hermansson <3tnica at gmail.com>
>> > # Date 1445625283 -7200
>> > #      Fri Oct 23 20:34:43 2015 +0200
>> > # Node ID 868fc6b3bf69be611118c597578e749c65698b8c
>> > # Parent  ee16fb0db905cfb858a929374cf623cdf1a0f9d3
>> > Most standard syslog facilicties do not expect hostname when logging to
>> > Unix Sockets.
>> > This patch removes hostname from syslog message when logging to Unix
>> Socket.
>> > Tested on rsyslog and syslog-ng
>> >
>> > diff -r ee16fb0db905 -r 868fc6b3bf69 src/core/ngx_syslog.c
>> > --- a/src/core/ngx_syslog.c     Tue Oct 20 21:28:38 2015 +0300
>> > +++ b/src/core/ngx_syslog.c     Fri Oct 23 20:34:43 2015 +0200
>> > @@ -219,9 +219,17 @@
>> >      ngx_uint_t  pri;
>> >
>> >      pri = peer->facility * 8 + peer->severity;
>> > -
>> > -    return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
>> > &ngx_cached_syslog_time,
>> > +#if (NGX_HAVE_UNIX_DOMAIN)
>> > +    if  (peer->server.sockaddr->sa_family == AF_UNIX) {
>> > +        return ngx_sprintf(buf, "<%ui>%V %V: ", pri,
>> > &ngx_cached_syslog_time,
>> > +                       &peer->tag);
>> > +    } else {
>> > +#endif
>> > +        return ngx_sprintf(buf, "<%ui>%V %V %V: ", pri,
>> > &ngx_cached_syslog_time,
>> >                         &ngx_cycle->hostname, &peer->tag);
>> > +#if (NGX_HAVE_UNIX_DOMAIN)
>> > +    }
>> > +#endif
>> >  }
>>
>> Hi Nils,
>>
>> we agree that there is some inconvenience with local syslog and are
>> considering adding configuration option to select desired behaviour
>> at this place.
>>
>> Note that there is a simple solution - to setup your syslog daemon
>> to expect hostname. See https://trac.nginx.org/nginx/ticket/677
>> and https://trac.nginx.org/nginx/ticket/783.
>>
>> Your suggested patch would breaks existing setups and make message
>> content depend on transport, what is undesirable.
>>
>> _______________________________________________
>> nginx-devel mailing list
>> nginx-devel at nginx.org
>> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>>
>
>
> _______________________________________________
> nginx-devel mailing list
> nginx-devel at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151029/fae4f101/attachment.html>

From vl at nginx.com  Thu Oct 29 10:20:15 2015
From: vl at nginx.com (Vladimir Homutov)
Date: Thu, 29 Oct 2015 13:20:15 +0300
Subject: Syslog Unix socket patch
In-Reply-To: <CAJp36wV4G66G-foEehiw+PtP7=UBsurmHX+Ev1qLJYtewyqrtQ@mail.gmail.com>
References: <CAJp36wVFRXofZ-A7OvvevxMZMZ2KdT2ZvCwDfJ-B-HO6yOim-A@mail.gmail.com>
 <20151027141439.GA29084@vlpc.nginx.com>
 <CAJp36wV4G66G-foEehiw+PtP7=UBsurmHX+Ev1qLJYtewyqrtQ@mail.gmail.com>
Message-ID: <20151029102014.GA30067@vlpc.nginx.com>

On Tue, Oct 27, 2015 at 08:21:46PM +0100, Nils Hermansson wrote:
> Created a new patch as disabling hostame as a configuration option.

I've commited a simpler patch that was ready some time ago:

http://hg.nginx.org/nginx/rev/a6a2016b8e31

thank you for efforts though!


From mdounin at mdounin.ru  Fri Oct 30 19:16:21 2015
From: mdounin at mdounin.ru (Maxim Dounin)
Date: Fri, 30 Oct 2015 19:16:21 +0000
Subject: [nginx] Fixed ngx_parse_time() out of bounds access (ticket #821).
Message-ID: <hg.4ccb37b04454.1446232581.6026610855610030274@dev.nginx.com>

details:   http://hg.nginx.org/nginx/rev/4ccb37b04454
branches:  
changeset: 6287:4ccb37b04454
user:      Maxim Dounin <mdounin at mdounin.ru>
date:      Fri Oct 30 21:43:30 2015 +0300
description:
Fixed ngx_parse_time() out of bounds access (ticket #821).

The code failed to ensure that "s" is within the buffer passed for
parsing when checking for "ms", and this resulted in unexpected errors when
parsing non-null-terminated strings with trailing "m".  The bug manifested
itself when the expires directive was used with variables.

Found by Roman Arutyunyan.

diffstat:

 src/core/ngx_parse.c |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diffs (12 lines):

diff --git a/src/core/ngx_parse.c b/src/core/ngx_parse.c
--- a/src/core/ngx_parse.c
+++ b/src/core/ngx_parse.c
@@ -188,7 +188,7 @@ ngx_parse_time(ngx_str_t *line, ngx_uint
             break;
 
         case 'm':
-            if (*p == 's') {
+            if (p < last && *p == 's') {
                 if (is_sec || step >= st_msec) {
                     return NGX_ERROR;
                 }


From gru at linuxiarz.pl  Sat Oct 31 22:51:38 2015
From: gru at linuxiarz.pl (=?UTF-8?Q?Mateusz_Gruszczy=C5=84ski?=)
Date: Sat, 31 Oct 2015 23:51:38 +0100
Subject: Opera issue
Message-ID: <787c0d6e-a9c2-4463-ad13-bec1660d3973@typeapp.com>

Hi everyone, I have a little trouble with the operation of HTTP / 2? Opera browser

For Mozilla FF and Chrome all OK. (?http://prntscr.com/8xlrln?)

In Opera I have an error in the style of "not received data." ->?http://prntscr.com/8xlr5j

VHost->?http://pastebin.com/CqJqnvmG
Server CNF ->?http://pastebin.com/QMtHN4a3
NGINX packets->?http://pastebin.com/MDKrErmH

Tested on a verified certificate and is the same thing.

Version Opera 32.0 (Windows 8.1)

Does anyone have a solution?

It may be for Opera to disable http/2 and the rest have included?

--?

Pozdrawiam || Best regards

Mateusz Gruszczy?ski
linuxiarz.pl

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx-devel/attachments/20151031/cdf37f27/attachment.html>