subversion

Checkout Tools
  • last updated 2 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1714640 is being indexed.

Merge the 'svnmover' prototype from the 'move-tracking-2' branch to trunk.

See the dev@ email thread "Merge 'svnmover' demo tool to trunk" started on

2015-11-10, archived at e.g. <http://mail-archives.apache.org/mod_mbox/subversion-dev/201511.mbox/%3CCAEcU=1YE5Ci02j_4aAeB_roS1X0H2QWPP3juu0exjVJ-FGCLyg@mail.gmail.com%3E>.

This is a reintegration (performed as an automatic merge) followed by

reverting the unwanted parts. As such, those unwanted parts (such as shim

insertions in several libraries) will be treated as 'record-only merged' and

will not be picked up by any subsequent automatic merge to trunk.

A brief description of the changes follows.

* build.conf

Adjust to build the new files.

* configure.ac

Adjust compiler warning flags to not warn about passing or returning a

structure, because some of the code does so.

* LICENSE

Add the licence text for the embedded 'linenoise' library.

* notes/move-tracking/README

New file, describing this work.

* subversion/include/private/svn_branch.h,

subversion/include/private/svn_branch_compat.h,

subversion/include/private/svn_branch_impl.h,

subversion/include/private/svn_branch_nested.h,

subversion/include/private/svn_branch_repos.h,

subversion/include/private/svn_element.h

New files.

* subversion/libsvn_delta/branch.c,

subversion/libsvn_delta/branch_compat.c,

subversion/libsvn_delta/branch_migrate.c,

subversion/libsvn_delta/branch_nested.c,

subversion/libsvn_delta/branch_repos.c,

subversion/libsvn_delta/element.c

New files.

* subversion/include/private/svn_cmdline_private.h

subversion/libsvn_subr/cmdline.c

(svn_cmdline__stdin_is_a_terminal,

svn_cmdline__stdout_is_a_terminal,

svn_cmdline__stderr_is_a_terminal): New.

* subversion/tests/cmdline/svnmover_tests.py

New file.

* subversion/tests/cmdline/svntest/actions.py

(run_and_verify_svnmover,

run_and_verify_svnmover2): New.

* subversion/tests/cmdline/svntest/main.py

(svnmover_binary,

run_svnmover): New.

(execute_tests): Initialize 'svnmover_binary'.

* subversion/tests/cmdline/svntest/wc.py

(State): Add 'rename' and 'from_eids' methods.

(StateItem): Add an 'eid' attribute.

* tools/dev/svnmover

New tool.

    • ?
    /trunk/subversion/libsvn_delta/element.c
    • ?
    /trunk/subversion/include/private/svn_branch_nested.h
    • ?
    /trunk/subversion/include/private/svn_branch_impl.h
    • ?
    /trunk/subversion/include/private/svn_branch_repos.h
    • ?
    /trunk/subversion/include/private/svn_branch_compat.h
    • ?
    /trunk/subversion/include/private/svn_branch.h
    • ?
    /trunk/subversion/include/private/svn_element.h
  1. … 11 more files in changeset.
On the 'move-tracking-2' branch: Restore an svn:eol-style property that I

accidentally deleted in r1626177.

* subversion/libsvn_ra_local/ra_plugin.c

Add svn:eol-style=native.

On the 'move-tracking-2' branch: Elide redundant subtree mergeinfo by using

'svn-mergeinfo-normalizer normalize'.

  1. … 6 more files in changeset.
On the 'move-tracking-2' branch: Move log-scanning code out of libsvn_client

and into svnmover, to localize the experimental source code as much as

possible.

* subversion/include/private/svn_client_private.h

(svn_repos_move_info_t,

svn_repos_move_info_create,

svn_client__format_move_chain_for_display,

svn_client__get_repos_moves): Move to svnmover.h.

* subversion/libsvn_client/update.c

(svn_repos_move_info_create,

svn_client__format_move_chain_for_display,

scan_moves_log_receiver_baton,

copy_info,

check_ancestry,

scan_moves_log_receiver,

svn_client__get_repos_moves): Move to scanlog.c.

(update_internal): Tweak commented-out code such that it could be compiled.

* tools/dev/svnmover/scanlog.c

New file, with contents moved from update.c.

* tools/dev/svnmover/svnmover.h

(svn_repos_move_info_t,

svn_repos_move_info_create,

svn_client__format_move_chain_for_display,

svn_client__get_repos_moves): Move from svn_client_private.h..

    • ?
    /branches/move-tracking-2/tools/dev/svnmover/scanlog.c
On the 'move-tracking-2' branch: Revert a test expectation change that

matched an earlier version of editor shims.

This reverts a part of r1615064.

* subversion/tests/cmdline/copy_tests.py

(url_to_non_existent_url_path): Tweak a path in an expected error message.

On the 'move-tracking-2' branch: Move a description of the work from

BRANCH-README to notes/move-tracking/README.

* BRANCH-README

Move most of the notes from here...

* notes/move-tracking/README

... to this new file.

    • ?
    /branches/move-tracking-2/notes/move-tracking/README
On the 'move-tracking-2' branch: Rename a file.

* subversion/libsvn_delta/migrate.c

Rename...

* subversion/libsvn_delta/branch_migrate.c

... to this. Also remove unwanted 'svn:mime-type' property.

On the 'move-tracking-2' branch: Move RA functionality out of libsvn_ra and

into 'svnmover', to localize the experimental source code as much as

possible.

* subversion/include/private/svn_ra_private.h

(svn_ra_load_branching_state,

svn_ra_get_commit_txn): Move declarations from here.

(svn_ra_fetch): Remove declaration. (Rename to 'fetch'; make private.)

* subversion/libsvn_ra/ra_loader.c

(svn_ra_get_commit_editor3): Remove insertion of compatibility shims.

(read_rev_prop,

write_rev_prop,

branch_revision_fetch_info,

txn_fetch_payloads,

branch_repos_fetch_info,

branch_get_mutable_state,

store_repos_info,

ccw_baton,

commit_callback_wrapper,

remap_commit_callback,

fb_baton,

svn_ra_fetch,

svn_ra_load_branching_state,

svn_ra_get_commit_txn): Move from here...

* tools/dev/svnmover/ra.c

... to this new file.

* tools/dev/svnmover/svnmover.h

(svn_ra_load_branching_state,

svn_ra_get_commit_txn): Move declarations to here.

    • ?
    /branches/move-tracking-2/tools/dev/svnmover/ra.c
On the 'move-tracking-2' branch: catch up to trunk@1714533.
  1. … 11 more files in changeset.
On the 'move-tracking-2' branch: Adjust documentation following r1714181.

Found by: rhuijben

* subversion/tests/cmdline/svntest/main.py

(run_svnmover): Adjust the doc string.

* STATUS:

Let's try this one more time :/

Correct revision from r1714313 to r1714314.

* STATUS:

Correct revision from r1714317 to r1714313.

* STATUS:

Add r1714317 and vote for it (svn_ra_svn_create_conn3 regression).

Add r1714358 and vote for it (overflow checks in svn_stringbuf_*).

Merge r1711346 from trunk:

* r1711346

Properly remove tempfiles in svnlook on diff errors such as EPIPE.

Justification:

Simple fix makes it far less likely that tempfiles are left behind

on the server.

Votes:

+1: rhuijben, stefan2, ivan

    • ?
    /branches/1.9.x/subversion/svnlook/svnlook.c
* STATUS: Vote for r1711346. Approving.

* STATUS: Vote for issue #4598 fix.

* subversion/libsvn_subr/cache-membuffer.c

(ensure_data_insertable_l2): Fix wording in comment. No functional change.

Fix svn_sort__array_delete for very large arrays.

The only way to trigger this in our current code is to have

mergeinfo with more than 1G ranges per path.

* subversion/libsvn_subr/sorts.c

(svn_sort__array_delete): Use the right overflow check.

Fix a number of potential overflow conditions on platforms where pointers

may be allocated very close to the end of address space, such as WoW64.

Instead of "if (base + x < max)", we must use "if (max - base > x)" to

prevent overflows under all circumstances.

There is no direct way to use these to trigger a segfault on purpose,

rather a user might experience random crashes. Due to the block sizes

we allocate and the fact that OS and RTL reserve the very top of the

address space, it is very unlikely to ever actually encounter an overflow

with valid lengths and offsets in SVN.

* subversion/libsvn_subr/cache-membuffer.c

(ensure_data_insertable_l2,

ensure_data_insertable_l1,

membuffer_cache_set_partial_internal): Fix the overflow check as

indicated above.

* subversion/libsvn_subr/compress.c

(svn__decode_uint): Same.

* subversion/libsvn_subr/prefix_string.c

(svn_prefix_string__create): Same.

* subversion/libsvn_subr/skel.c

(explicit_atom): Same.

* subversion/libsvn_subr/spillbuf.c

(svn_spillbuf__write): Same.

* subversion/libsvn_subr/subst.c

(translate_chunk): Same.

* subversion/libsvn_delta/svndiff.c

(write_handler): Code formatting fixes. No functional change.

Fix the overflow / truncating handling handling of svn_stringbuf_remove

and svn_stringbuf_replace. The API allows for deleted region to extend

beyond the current string; APR_SIZE_MAX in particular is a valid length

for the region to remove.

Note that all calls within our code use properly limited calls to these

functions, so they never may cause overflows. 3rd party callers might

be affected, though.

* subversion/libsvn_subr/string.c

(svn_stringbuf_remove,

svn_stringbuf_replace): Correct the removal length limiting code.

* subversion/tests/libsvn_subr/string-test.c

(test_stringbuf_remove,

test_stringbuf_replace): Add test cases for the fixed conditions.

* subversion/libsvn_subr/cache-membuffer.c

(svn_cache__create_membuffer_cache): Add a paranoia parameter check.

Fix capacity check in the membuffer cache's prefix pool.

The current implementation used only about 1% of its capacity.

* subversion/libsvn_subr/cache-membuffer.c

(prefix_pool_get_internal): Actually check byte usage against bytes

limit - not entry count limit.

For ra-svn client requests that exceed the newly introduced limit,

usually there is only one huge string component, e.g. some large

property value, that causes the problem. Detect them early such

that we might not even need to transfer the whole data up to the

limit before giving up.

* subversion/libsvn_ra_svn/marshal.c

(read_string): Exit as soon as we know we will exceed the configured

size limit.

* subversion/libsvn_ra_svn/marshal.c

(read_string): Clarify a comment. No functional change.

Following up on r1714330 and r1714333, fix deprecation warnings.

* subversion/libsvn_ra_svn/client.c

(handle_child_process_error,

make_tunnel,

open_session): Call the latest API version but don't set any limits

because the client already has cancellation support

if needed.

Following up on r1714330, implement a similar limit to svnserve responses.

If --max-response-size is specified and a response exceeds that limit,

it gets truncated and the connetion will be reset. This allows us to limit

the server load caused by "don't do that" type of requests like exporting

the root of the repository instead of some project branch.

By default, this limit is 0, i.e. inactive.

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SVN_RESPONSE_SIZE): New error code.

* subversion/include/svn_ra_svn.h

(svn_ra_svn_create_conn5): Add another limit parameter.

* subversion/libsvn_ra_svn/ra_svn.h

(svn_ra_svn_conn_st): Add fields for the new data counter and its limit.

* subversion/libsvn_ra_svn/deprecated.c

(svn_ra_svn_create_conn4): Update.

* subversion/libsvn_ra_svn/marshal.c

(svn_ra_svn_create_conn5): Initialize the new struct elements.

(svn_ra_svn__reset_command_io_counters): One more counter to reset.

(check_io_limits): One more limit to check.

(writebuf_output): Count outgoing data and enforce limits.

* subversion/svnserve/server.h

(serve_params_t): Add field for the new --max-response-size option.

* subversion/svnserve/serve.c

(serve_interruptable): Pass the new option to the bumped API.

* subversion/svnserve/svnserve.c

(SVNSERVE_OPT_MAX_RESPONSE): Declare the new option.

(svnserve__options): Define and document the new option.

(sub_main): Handle the new option and pass it to the bumped API.

Add the equivalent of LimitXMLRequestBody to svnserve.

The idea is simple, whenever we fill our receive buffer, we update the sum

total and compare it to some limit. Reset the counter sum at each new

command / request coming in.

If a client request exceeds the --max-request-size parameter given to

svnserve (16MB by default, twice the httpd default), the processing gets

terminated and the connection will be closed. The latter is necessary

because the protocol is stateful and we just skipped / ignored a potential

state transition.

As a result, the memory usage of a threaded server is now bound to approx.

(max-request-size + 4M) x max-threads even in high-load scenarios. On the

flip side, propsets are limited to around 15M per property by default.

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SVN_REQUEST_SIZE): New error code.

* subversion/include/svn_ra_svn.h

(svn_ra_svn_create_conn5): Bumped API, adding the new limit parameter.

(svn_ra_svn_create_conn4): Deprecate.

* subversion/libsvn_ra_svn/ra_svn.h

(svn_ra_svn_conn_st): Add fields for the data counter and its limit.

(svn_ra_svn__reset_command_io_counters): Declare a function to reset the

counter - to be called before

each new command.

* subversion/libsvn_ra_svn/deprecated.c

(svn_ra_svn_create_conn4): Implement in terms of the new API.

* subversion/libsvn_ra_svn/marshal.c

(svn_ra_svn_create_conn5): Implement like the predecessor but init the

new struct elements as well.

(svn_ra_svn__reset_command_io_counters): Implement new private API.

(check_io_limits): New function performing the actual error detection.

(readbuf_input): Count incoming data and enforce limits.

(svn_ra_svn__has_command): Be sure to count I/O per command.

(svn_ra_svn__handle_command): Same. Also handle the case that we truncated

I/O and are now in a potentially inconsistent

state.

* subversion/libsvn_ra_svn/editorp.c

(svn_ra_svn_drive_editor2): Limit the request size separately for each

editor command - not the whole editor drive.

* subversion/svnserve/server.h

(serve_params_t): Add field for the new --max-request-size option.

* subversion/svnserve/serve.c

(serve_interruptable): Pass the new option to the bumped API.

* subversion/svnserve/svnserve.c

(MAX_REQUEST_SIZE): Define the default value for the new option.

(SVNSERVE_OPT_MAX_REQUEST): Declare the new option.

(svnserve__options): Define and document the new option.

(sub_main): Handle the new option and pass it to the bumped API.

Improve pool usage in svnserve and ra_svn.

All callers to the protocol parser now (indirectly) use a short-lived

scratch pool to hold the request data.

* subversion/libsvn_ra_svn/client.c

(ra_svn_get_locations): Use an ITERPOOL as scratch within the loop.

* subversion/svnserve/serve.c

(internal_auth_request): Same.

Some code cleanup in svnserve.

* subversion/svnserve/serve.c

(auth): Put the return parameter in front of all others.

Expliciyly mark the pool as a SCRATCH_POOL.

(internal_auth_request): Update caller.