kotkov in httpd

On the 2.4.x branch: Propose the mod_brotli/mod_deflate 304 handling

fix (r1843242) for backport.

mod_brotli, mod_deflate: Restore the separate handling of 304 Not Modified

responses allowing these modules to properly set or fix-up the response

headers such as Vary or ETag.

This change follows up on r1837056 that disabled that special handling and

thus resulted in a potential violation of RFC7232, 4.1:

The server generating a 304 response MUST generate any of the following

header fields that would have been sent in a 200 (OK) response to the

same request: Cache-Control, Content-Location, Date, ETag, Expires,

and Vary.)


mpm_winnt: Do not redefine the standard CONTAINING_RECORD() macro

in child.c.

This definition has been added in — perhaps,

because not every versions of SDK contained it at that time.

But since then, the macro has been available starting from Windows 2000


and any available version of Windows SDK now should also contain it.

mpm_winnt: Remove an obsolete comment in child.c explaining why the

declarations of the structures and functions to access the completion

contexts reside in a header file.

This no longer holds, as all the necessary functions and structures are

located in the single .c file (child.c).

mpm_winnt: Tweak the names of the variables in child.c which are used to

represent a queue of the completion contexts.

Starting from r1801655, the "queue" isn't really a queue, as all the

access happens with a LIFO order. So, instead of that, call it a "pool

of completion contexts", adjust names of all relevant variables and

tweak the comments.

This patch changes

- qlock to ctxpool_lock,

- qhead to ctxpool_head, and

- qwait_event to ctxpool_wait_event.

mpm_winnt: Tweak the listener shutdown code to use a separate event

instead of the global variable (shutdown_in_progress).

This change has two purposes. First of all, it makes the listener threads

which are blocked waiting for a completion context exit immediately during

shutdown. Previously, such threads would only check for exit every second.

The second reason for this change is to put the child_main() function in

charge of controlling the listeners life cycle. Previously, such relation

was circumvented by the fact that the listeners were also waiting for the

global child exit_event. With the new separate listener_shutdown_event,

only the child_main() function is responsible for shutting down the

listeners, and I think that this makes the code a bit clearer.

All the original behavior, including the special APLOG_DEBUG diagnostic

message when we fail to acquire a free completion context in 1 second,

is kept unchanged.

mpm_winnt: Following up on r1801655, add a comment that explains the

reason to choose the LIFO processing order for completion contexts.

It would be better to keep this important information in the code, instead

of just having it in the log message.

mpm_winnt: Advertise support for preshutdown notifications in the service,

and perform shutdown in respond to SERVICE_CONTROL_PRESHUTDOWN.

The pure shutdown notification leaves a small amount of time for the service

to finish (and the allowed amount of time has been shrinking with every new

version of Windows), and handling only it increases the chance of the process

being killed by SCM, instead of gracefully shutting down. Handling the

preshutdown control code extends this period, and increases the chances of

finishing everything properly when the machine is rebooted or shut down.


Please note that although the preshutdown notifications are available only

starting from Windows Vista, the code is compatible with the previous versions

of Windows, since the SCM ignores unknown SERVICE_ACCEPT codes, and will

still send an ordinary SERVICE_CONTROL_SHUTDOWN under old Windows


mpm_winnt: Remove unused values of the io_state_e enum.

Submitted By: Ivan Zhakov <ivan {at}>

mpm_winnt: Remove a duplicated comment in the child_main() function.

mpm_winnt: Use a LIFO stack instead of a FIFO queue to hold unused

completion contexts, as that may significantly reduce the memory usage.

This simple change can have a noticeable impact on the amount of memory

consumed by the child process in various cases. Every completion context

in the queue has an associated allocator, and every allocator has it's

ap_max_mem_free memory limit which is not given back to the operating

system. Once the queue grows, it cannot shrink back, and every allocator

in each of the queued completion contexts keeps up to its max_free amount

of memory. The queue can only grow when a server has to serve multiple

concurrent connections at once.

With that in mind, consider a case with a server that doesn't encounter many

concurrent connections most of the time, but has occasional spikes when

it has to serve multiple concurrent connections. During such spikes, the

size of the completion context queue grows.

The actual difference between using LIFO and FIFO orders shows up after

such spikes, when the server is back to light load and doesn't see a lot

of concurrency. With FIFO order, every completion context in the queue

will be used in a round-robin manner, thus using *every* available allocator

one by one and ultimately claiming up to (N * ap_max_mem_free memory) from

the OS. With LIFO order, only the completion contexts that are close to

the top of the stack will be used and reused for subsequent connections.

Hence, only a small part of the allocators will be used, and this can

prevent all other allocators from unnecessarily acquiring memory from

the OS (and keeping it), and this reduces the overall memory footprint.

Please note that this change doesn't affect the worst case behavior, as

it's still (N * ap_max_mem_free memory), but tends to behave better in

practice, for the reasons described above.

Another thing worth considering is the new behavior when the OS decides

to swap out pages of the child process, for example, in a close-to-OOM

condition. Handling every new connection after the swap requires the OS

to load the memory pages for the allocator from the completion context that

is used for this connection. With FIFO order, the completion contexts are

used one by one, and this would cause page loads for every new connection.

With LIFO order, there will be almost no swapping, since the same completion

context is going to be reused for subsequent new connections.

mpm_winnt: Drop the APLOG_DEBUG diagnostic saying how many thread

are blocked on the I/O completion port during the shutdown.

Prior to r1801635, the shutdown code required to know the amount of blocked

threads, as it has been dispatching the same amount of completion packets.

But this no longer holds, and the only reason why we maintain the

corresponding g_blocked_threads variable is because of this debug

diagnostic message.

Drop it in order to reduce complexity of the quite critical code in the

winnt_get_connection() function and to reduce the amount of global


mpm_winnt: Remove an unnecessary Sleep() in the winnt_accept() function.

This sleep occured in a situation when:

- We don't have a free completion context in the queue

- We can't add one, as doing so would exceed the max_num_completion_contexts

limit (all worker threads are busy)

- We have exceeded a 1 second timeout while waiting for it

In this case, the Sleep() call is unnecessary, as there is no intermittent

failure that can be waited out, but rather than that, it's an ordinary

situation with all workers being busy. Presumably, calling Sleep() here

can be even considered harmful, as it affects the fairness between the

listeners that are blocked waiting for the completion context.

So, instead of calling Sleep() just check for the possible shutdown and

immediately retry acquiring a completion context. If all worker threads

are still busy, the retry will block in the same WaitForSingleObject() call,

which is fine.

mpm_winnt: Simplify the shutdown code that was waiting for multiple worker

thread handles in batches.

Starting from r1801636, there is no difference between ending the wait with

one or multiple remaining threads. This is because we terminate the process

if at least one thread is still active when we hit a timeout.

Therefore, instead of making an effort to evenly distribute and batch the

handles with WaitForMultipleObjects(), we could just start from one end,

and wait for one thread handle at a time.

mpm_winnt: Avoid using TerminateThread() in case the shutdown routine

hits a timeout while waiting for the worker threads to exit.

Using TerminateThread() can have dangerous consequences such as deadlocks —

say, if the the thread is terminated while holding a lock or a heap lock

in the middle of HeapAlloc(), as these locks would not be released.

Or it can corrupt the application state and cause a crash.


Rework the code to call TerminateProcess() in the described circumstances

and leave the cleanup to the operating system.

mpm_winnt: Make the shutdown faster by avoiding unnecessary Sleep()'s

when shutting down the worker threads.

Previously, the shutdown code was posting an amount of I/O completion

packets equal to the amount of the threads blocked on the I/O completion

port. Then it would Sleep() until all these threads "acknowledge" the

completion packets by decrementing the global amount of blocked threads.

A better way would be to send the number of IOCP_SHUTDOWN completion

packets equal to the total amount of threads and immediately proceed to

the next step. There is no need to block until the threads actually receive

the completion, as the shutdown process includes a separate step that waits

until the threads exit, and the new approach avoids an unnecessary delay.

mpm_winnt: Following up on r1801144, use the new accept_filter_e enum

values in a couple of missed places in winnt_accept().

mpm_winnt: Fix typo in the logged message in winnt_get_connection().

mpm_winnt: Refactor the mpm_get_completion_context() function so that it

would return a proper apr_status_t instead of yielding the result via the

*timeout out variable.

This makes the calling side easier to follow by avoiding an additional

layer of if's.

mpm_winnt: Remove an unnecessary retry after receiving a non-timeout failure

from the mpm_get_completion_context() function.

Currently, the only possible reasons why mpm_get_completion_context() could

fail are real errors such as being unable to WaitForSingleObject(), allocate

memory or create an event. Retrying under such circumstances doesn't make

sense, and could be as well considered harmful.

mpm_winnt: Factor out a helper function to parse the type of an accept

filter and use an appropriate enum for it.

This makes the code in winnt_accept() a bit easier to follow. As a minor

side effect, it also fixes a small bug where the "unrecognized AcceptFilter

'%s'" log entry would always contain "none" instead of the actually

unrecognized kind of the accept filter.

mpm_winnt: Don't forget to close the I/O completion port as part of the

cleanup in the child process.

On the 2.4.x branch: Propose the mod_brotli Makefile fixes (r1761824,

r1771789, r1771827, r1779111) for backport.

Create a backport branch with r1761824, r1771789, r1771827 and r1779111

applied to 2.4.x.

These are the mod_brotli related Makefile changes that didn't make it

into the original backport proposal merged in r1791231. The lack of these

changes causes a failing Unix build in my environment if mod_brotli is not

being built. Another issue is that by default the CMakeLists.txt file

refers to invalid library filenames.

Shortlog of the changes:

r1761824: Unbreak building other filter modules without libbrotlienc.

r1771789: Rewrite the autoconf script in a, hopefully, less convoluted way.

This lays the groundwork to simplify the switch to the official Brotli


r1771827: Update makefiles to use the library layout of the official

Brotli repository.

r1779111: Update makefile to cope with the pkg-config layout change


Also see

mod_brotli: Fix leftovers from mod_deflate or incorrect directives in

the "Serving pre-compressed content" section of the docs.

Generally speaking, this section would benefit from a rewrite pointing

out how to configure a mod_deflate + mod_brotli configuration with

precompressed contents, but for now at least fix the mistakes in the


mod_brotli: Tweak the descriptions of the directives provided by mod_brotli

in the documentation (BrotliCompressionQuality, BrotliCompressionWindow,

BrotliCompressionMaxInputBlock, BrotliAlterETag).

mod_brotli: Nuke the section about input decompression using mod_brotli

in the documentation.

Currently, mod_brotli only allows dynamic output compression, and doesn't

have the server-side decompression capability.

mod_brotli: Properly describe the "no-brotli" environment variable in

the documentation.

The previous description explained the semantics of this variable as

if it has been a (non-existing) "force-brotli".

mod_brotli: Remove incorrect references to mod_deflate in the documentation.

mod_brotli: Comment on the default choice (0) for BROTLI_PARAM_LGBLOCK.