Yann Ylavic

Follow up to r1879449: yet better MPM poll callback API.

Let pass a const pfds to the MPM, for it to make a copy on the given pool

as needed.

mpm_common: add pool argument to mpm_register_poll_callback[_timeout] hooks.

This is cleaner API than using pfds->pool implicitely.

MAJOR bump but reusing the existing hooks (with an API/ABI breakage) because

it's trunk material only.

mod_proxy_wstunnel: follow up to r1879418: handle first async lifetime too.

Create the dedicated pfds and subpool in proxy_wstunnel_request() too, for

the first call to ap_mpm_register_poll_callback_timeout().

While at it, add comments about why we need the dedicated pfds/subpool.

mod_proxy_http: follow up to r1879419: clarify poll callback pds/subpool.

Comments about why we need a dedicated pfds and its subpool for

ap_mpm_register_poll_callback_timeout().

mpm_common: remove ap_mpm_unregister_poll_callback().

It's now called automatically by mpm_event and anyway can't be called safely

outside the MPM code without racing.

MAJOR bump.

Follow up to r1879419: CHANGES entry.
mod_proxy_http: handle async tunneling of Upgrade(d) protocols.

When supported by the MPM (i.e. "event"), provide async callbacks and let

them be scheduled by ap_mpm_register_poll_callback_timeout(), while the

handler returns SUSPENDED.

The new ProxyAsyncDelay directive (if positive) enables async handling,

while ProxyAsyncIdleTimeout determines the timeout applied on both ends

while tunneling.

Github: closes #126

mod_proxy_wstunnel: avoid leaks on tunnel->pfds->pool.

Since event_register_poll_callback_ex() allocates its data on pfds->pool,

we need a subpool to be cleared at each proxy_wstunnel_callback() call.

mpm_event: poll callbacks fixes and improvements.

server/mpm_fdqueue.h;

Rename "remove" field to "pfds" in timer_event_t.

server/mpm/event/event.c:

update_reqevents_from_sense():

New helper to update pfd->reqevents according to the given cs->sense

for CONN_STATE_WRITE_COMPLETION, reusable in process_socket() and

event_resume_suspended().

event_resume_suspended():

Process lingering close if given cs->state = CONN_STATE_LINGER.

Call notify_suspend() before entering CONN_STATE_WRITE_COMPLETION.

event_register_poll_callback_ex():

Don't poll pfds with reqevents == 0.

listener_thread():

Run event_cleanup_poll_callback to both remove the registered pfds

and leave pfds->pool in a consistent state.

Process users callabacks after all PT_USER batons have been collected

in the result pfds loop, otherwise we might race with the callbacks

within the loop if multiple events/sockets concern the same baton, and

crash if pfds->pool is cleared.

mod_ssl: release coalesced data when called from ap_filter_output_pending().

The purpose of ap_filter_output_pending() is to flush pending data, so

ssl_io_filter_coalesce() should honor that.

This allows mod_proxy to not care about mod_ssl coalescing filters when

tunneling between connections.

mod_proxy: reindent ap_proxy_transfer_between_connections() after r1879401.

No functional change.

mod_proxy: improved and reentrant tunneling loop.

modules/proxy/mod_proxy.h:

Rename AP_PROXY_TRANSFER_SHOULD_YIELD to AP_PROXY_TRANSFER_YIELD_PENDING

and add AP_PROXY_TRANSFER_YIELD_MAX_READS.

modules/proxy/mod_proxy_http.c:

modules/proxy/mod_proxy_wstunnel.c:

Removing of reqtimeout filter is now handled by ap_proxy_tunnel_create().

modules/proxy/proxy_util.c:

ap_proxy_transfer_between_connections():

Reorganize loop to break out early.

When AP_PROXY_TRANSFER_YIELD_PENDING, if !ap_filter_should_yield() we

still need to run and check ap_filter_output_pending() since it may

release pending data.

When AP_PROXY_TRANSFER_YIELD_MAX_READS, stop the loop after too much

reads (PROXY_TRANSFER_MAX_READS = 10000) to release the thread and

give the caller a chance to schedule the other direction.

Don't return APR_INCOMPLETE when it comes from an incomplete body

detected by ap_http_filter().

ap_proxy_tunnel_create():

Start with POLLOUT on both directions so that any pending output data

is flushed first.

ap_proxy_tunnel_run():

Remove re-init/clear of the pollset for each call so that the function

is reentrant.

Handle POLLOUT before POLLIN so that we can read in the same pass once

all buffered output data are flushed, using ap_filter_input_pending()

to drain buffered input data.

This is preparatory patch for async websocket tunneling is mod_proxy_http.

tab [skip ci].
2.4.x patch Eric's proposal, with latest changes [skip ci].
Follow up to r1877955: don't reuse the connection for mixed C-L / T-E requests

Disable keepalive on the connection if we received both Content-Length and

chunked Transfer-Encoding in the request, to avoid confusion with front

intermediaries and potential further request/response splitting.

This is what we do already for mod_proxy backend connections in the same case.

While at it, replace draft httpbis links with final RFC7230's.

Follow up to r1877955: always validate the Content-Length

even if it is to be ignored because of RFC7230 section 3.3.3 requirements.

Vote + promote [skip ci].
Revert r1879361: 2.4.x material only.
Update proposal [skip ci].
mod_proxy: unfail mixed ProxyPass/<Proxy> and ProxyPassMatch/<ProxyMatch>.

It is not a failure in current 2.4.x, so to ease backport and to avoid compat

breakage simply warn about the second directive being ignored.

This commit can be reverted in trunk if we want next versions to fail in this

case.

[Reverted by r1879363]

Follow up to r1879235: fill APLOGNO().

Follow up to r1879080 and r1879137: servlet-normalize r->uri if matched.

If a ProxyPass mapping=servlet matches (in pre_trans hook), update r->uri with

the servlet normalization so that later <Location> or any dir context match

does not have to handle potential path parameters.

Backported [skip ci].
Merge r1878280 from trunk:

mod_proxy_http: don't strip EOS when spooling request body to file.

To prevent stream_reqbody() from sending the FILE and FLUSH bucket in separate

brigades, and thus apr_file_setaside() to trigger if network congestion occurs

with the backend, restore the EOS in spool_reqbody_cl() which was stripped

when spooling the request body to a file.

Until APR r1878279 is released (and installed by users), apr_file_setaside()

on a temporary file (mktemp) will simply drop the file cleanup, leaking the

fd and inode..

This fixes BZ 64452.

Submitted by: ylavic

Reviewed by: ylavic, jorton, rpluem

Backported [skip ci].
Merge r1879179, r1879180 from trunk:

EVP_PKEY_up_ref(): fix ref count locking type for proxy EVP pkey

When enabling client authentication for proxy (SSLProxyMachineCertificateFile),

the client certificate callback function ssl_callback_proxy_cert uses another

reference count locking type then one that is used by the caller function when

trying to free the private key afterwards by using EVP_PKEY_free.

This can lead to a race-condition on pkey->references resulting in a double

free error.

On my system, the error occurs sporadically when threaded health checking

(mod_watchdog) forces two threads competing for the client's private key.

For example, see following two backtraces of a coredump where thread 1 and

thread 15 both run into CRYPTO_free(). Actually, the private key should never

be freed during run-time nor should two threads ever enter CRYPTO_free()

concurrently.

(gdb) t 1

[Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))]

#0 0xf7f3f329 in __kernel_vsyscall ()

(gdb) bt

#0 0xf7f3f329 in __kernel_vsyscall ()

#1 0xf7cec9e7 in raise () from /lib32/libc.so.6

#2 0xf7cedfb9 in abort () from /lib32/libc.so.6

#3 0xf7d2a14d in ?? () from /lib32/libc.so.6

#4 0xf7d2fd27 in ?? () from /lib32/libc.so.6

#5 0xf7d3047d in ?? () from /lib32/libc.so.6

#6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434

#7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406

#8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475

#9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426

#10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008

#11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832

#12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231

#13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008

#14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245

#15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760

#16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590

#17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728

#18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590

#19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664

#20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806

#21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870

#22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266

#23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142

#24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0

#25 0xf7da862e in clone () from /lib32/libc.so.6

(gdb) t 15

[Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))]

#0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6

(gdb) bt

#0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6

#1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6

#2 0xf7cd89ab in ?? () from /lib32/libc.so.6

#3 0xf7d2a148 in ?? () from /lib32/libc.so.6

#4 0xf7d2fd27 in ?? () from /lib32/libc.so.6

#5 0xf7d3047d in ?? () from /lib32/libc.so.6

#6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434

#7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406

#8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475

#9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426

#10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008

#11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832

#12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231

#13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008

#14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245

#15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760

#16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590

#17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728

#18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590

#19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664

#20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806

#21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870

#22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266

#23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142

#24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0

#25 0xf7da862e in clone () from /lib32/libc.so.6

Many thanks to Armin for finding this.

Github: closes #129

Submitted by: Armin Abfalterer (arminabf)

Reviewed by: ylavic

Follow up to r1879179: CHANGES entry.

Reviewed by: ylavic, jorton, rpluem

Propose fix for gh #129 [skip ci].
Follow up to r1879179: CHANGES entry.
EVP_PKEY_up_ref(): fix ref count locking type for proxy EVP pkey

When enabling client authentication for proxy (SSLProxyMachineCertificateFile),

the client certificate callback function ssl_callback_proxy_cert uses another

reference count locking type then one that is used by the caller function when

trying to free the private key afterwards by using EVP_PKEY_free.

This can lead to a race-condition on pkey->references resulting in a double

free error.

On my system, the error occurs sporadically when threaded health checking

(mod_watchdog) forces two threads competing for the client's private key.

For example, see following two backtraces of a coredump where thread 1 and

thread 15 both run into CRYPTO_free(). Actually, the private key should never

be freed during run-time nor should two threads ever enter CRYPTO_free()

concurrently.

(gdb) t 1

[Switching to thread 1 (Thread 0xb2cfbb40 (LWP 16054))]

#0 0xf7f3f329 in __kernel_vsyscall ()

(gdb) bt

#0 0xf7f3f329 in __kernel_vsyscall ()

#1 0xf7cec9e7 in raise () from /lib32/libc.so.6

#2 0xf7cedfb9 in abort () from /lib32/libc.so.6

#3 0xf7d2a14d in ?? () from /lib32/libc.so.6

#4 0xf7d2fd27 in ?? () from /lib32/libc.so.6

#5 0xf7d3047d in ?? () from /lib32/libc.so.6

#6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434

#7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406

#8 0x08463917 in ssl3_send_client_certificate (s=0xad21f070) at s3_clnt.c:3475

#9 0x0845d62c in ssl3_connect (s=0xad21f070) at s3_clnt.c:426

#10 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008

#11 0x0846f9c8 in ssl23_get_server_hello (s=0xad21f070) at s23_clnt.c:832

#12 0x0846ea45 in ssl23_connect (s=0xad21f070) at s23_clnt.c:231

#13 0x08484213 in SSL_connect (s=0xad21f070) at ssl_lib.c:1008

#14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d3f450) at ssl_engine_io.c:1245

#15 0x08263ba6 in ssl_io_filter_output (f=0xb4d3f480, bb=0xacc079a0) at ssl_engine_io.c:1760

#16 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f480, bb=0xacc079a0) at util_filter.c:590

#17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d3f468, bb=0xacc079a0) at ssl_engine_io.c:1728

#18 0x080ea2c9 in ap_pass_brigade (next=0xb4d3f468, bb=0xacc079a0) at util_filter.c:590

#19 0x08251658 in hc_send (r=0xacc069b0, out=0x8c25ec8 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xacc079a0) at mod_proxy_hcheck.c:664

#20 0x08251eb3 in hc_check_http (baton=0xacc068d8) at mod_proxy_hcheck.c:806

#21 0x08252653 in hc_check (thread=0x8cc6b10, b=0xacc068d8) at mod_proxy_hcheck.c:870

#22 0x08383185 in thread_pool_func (t=0x8cc6b10, param=0x8c245e0) at misc/apr_thread_pool.c:266

#23 0x083baef6 in dummy_worker (opaque=0x8cc6b10) at threadproc/unix/thread.c:142

#24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0

#25 0xf7da862e in clone () from /lib32/libc.so.6

(gdb) t 15

[Switching to thread 15 (Thread 0xb44feb40 (LWP 16049))]

#0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6

(gdb) bt

#0 0xf7dd90a5 in _dl_addr () from /lib32/libc.so.6

#1 0xf7db610c in backtrace_symbols_fd () from /lib32/libc.so.6

#2 0xf7cd89ab in ?? () from /lib32/libc.so.6

#3 0xf7d2a148 in ?? () from /lib32/libc.so.6

#4 0xf7d2fd27 in ?? () from /lib32/libc.so.6

#5 0xf7d3047d in ?? () from /lib32/libc.so.6

#6 0x08499c70 in CRYPTO_free (str=0x93376b0) at mem.c:434

#7 0x084cc063 in EVP_PKEY_free (x=0x93376b0) at p_lib.c:406

#8 0x08463917 in ssl3_send_client_certificate (s=0xacf1baa0) at s3_clnt.c:3475

#9 0x0845d62c in ssl3_connect (s=0xacf1baa0) at s3_clnt.c:426

#10 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008

#11 0x0846f9c8 in ssl23_get_server_hello (s=0xacf1baa0) at s23_clnt.c:832

#12 0x0846ea45 in ssl23_connect (s=0xacf1baa0) at s23_clnt.c:231

#13 0x08484213 in SSL_connect (s=0xacf1baa0) at ssl_lib.c:1008

#14 0x08261e73 in ssl_io_filter_handshake (filter_ctx=0xb4d37430) at ssl_engine_io.c:1245

#15 0x08263ba6 in ssl_io_filter_output (f=0xb4d37460, bb=0xad101588) at ssl_engine_io.c:1760

#16 0x080ea2c9 in ap_pass_brigade (next=0xb4d37460, bb=0xad101588) at util_filter.c:590

#17 0x08263b07 in ssl_io_filter_coalesce (f=0xb4d37448, bb=0xad101588) at ssl_engine_io.c:1728

#18 0x080ea2c9 in ap_pass_brigade (next=0xb4d37448, bb=0xad101588) at util_filter.c:590

#19 0x08251658 in hc_send (r=0xad100598, out=0x8c25898 "GET /hcheck HTTP/1.0\r\nHost: XXX\r\n\r\n", bb=0xad101588) at mod_proxy_hcheck.c:664

#20 0x08251eb3 in hc_check_http (baton=0xad1004c0) at mod_proxy_hcheck.c:806

#21 0x08252653 in hc_check (thread=0x8cc6ab0, b=0xad1004c0) at mod_proxy_hcheck.c:870

#22 0x08383185 in thread_pool_func (t=0x8cc6ab0, param=0x8c245e0) at misc/apr_thread_pool.c:266

#23 0x083baef6 in dummy_worker (opaque=0x8cc6ab0) at threadproc/unix/thread.c:142

#24 0xf7ec615f in start_thread () from /lib32/libpthread.so.0

#25 0xf7da862e in clone () from /lib32/libc.so.6

Many thanks to Armin for finding this.

Github: closes #129

Submitted by: Armin Abfalterer (arminabf)

Reviewed by: ylavic

Vote, promote.