libsvn_delta

Checkout Tools
  • last updated 6 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Improve backward compatibility for svn_delta_path_driver2().

This ensures the input array is not modified, and changes the behaviour to

be less surprising.

* subversion/include/svn_delta.h,

subversion/libsvn_delta/deprecated.c

(svn_delta_path_driver2): Add back slash prefixes if any, rather than the

first, of the inputs had one. Duplicate the array if modifying it.

  1. … 1 more file in changeset.
In the updated delta editor path driver, ensure paths are relpaths.

This also fixes a bug with '/'-prefixed paths in svn_delta_path_driver2(). A

path given as the string "/" was handled as a regular path (first open the

editor root, then call the callback passing it the root parent-dir-baton)

rather than as a root path.

* subversion/include/svn_delta.h

(svn_delta_path_driver_cb_func2_t,

svn_delta_path_driver3,

svn_delta_path_driver_step): Declare that paths are relpaths.

(svn_delta_path_driver2): Document that a slash prefix is allowed.

* subversion/libsvn_delta/deprecated.c

(path_driver_2_to_3_baton_t,

path_driver_2_to_3_func,

svn_delta_path_driver2): Implement backward compatibility for a slash

prefix.

* subversion/libsvn_delta/path_driver.c

(svn_delta_path_driver3): Rename variables to 'relpath', for clarity.

(svn_delta_path_driver_step): Assert the path is a relpath, remove support

for a slash prefix, and rename variables.

  1. … 1 more file in changeset.
Minor cleanup: extract a helper function.

* subversion/libsvn_delta/path_driver.c

(push_dir_stack_item): New.

(open_dir): Use it here.

(svn_delta_path_driver_step): Use it twice here.

Upgrade uses of the editor path driver callback, following r1852526.

* subversion/libsvn_client/add.c

(path_driver_cb_func): Take the editor and edit-baton as direct parameters.

(mkdir_urls): Upgrade to svn_delta_path_driver3.

* subversion/libsvn_client/commit_util.c

(item_commit_baton): Remove the editor and edit-baton from here.

(do_item_commit): Take the editor and edit-baton as direct parameters.

(svn_client__do_commit): Upgrade to svn_delta_path_driver3.

* subversion/libsvn_client/copy.c

(path_driver_cb_baton): Remove the editor and edit-baton from here.

(path_driver_cb_func): Take the editor and edit-baton as direct parameters.

(repos_to_repos_copy): Upgrade to svn_delta_path_driver3.

* subversion/libsvn_delta/branch_compat.c

(apply_change): Take the editor and edit-baton as direct parameters.

(drive_changes): Upgrade to svn_delta_path_driver3.

* subversion/libsvn_delta/compat.c

(apply_change): Take the editor and edit-baton as direct parameters.

(drive_changes): Upgrade to svn_delta_path_driver3.

* subversion/libsvn_repos/replay.c

(path_driver_cb_baton): Remove the editor and edit-baton from here.

(path_driver_cb_func): Take the editor and edit-baton as direct parameters.

(svn_repos_replay2): Upgrade to svn_delta_path_driver3.

  1. … 4 more files in changeset.
Make the editor path driver callback a little easier to use.

Add 'editor' and 'edit_baton' parameters to the path driver callback. All

but trivial callback implementations need these, and they otherwise had to

be passed through the callback baton, which was straightforward but more

trouble than it need be.

* subversion/include/svn_delta.h

(svn_delta_path_driver_cb_func2_t,

svn_delta_path_driver3): New.

* subversion/libsvn_delta/path_driver.c

(svn_delta_path_driver3): Rename from 'svn_delta_path_driver2'.

(svn_delta_path_driver_state_t,

svn_delta_path_driver_start): Change the callback function type.

(svn_delta_path_driver_step): Pass the extra parameters.

* subversion/libsvn_delta/deprecated.c

(path_driver_2_to_3_baton_t,

path_driver_2_to_3_func,

svn_delta_path_driver2): New.

* subversion/libsvn_client/shelf.c

(path_driver_cb_func): Take 'editor' and 'edit_baton' as direct parameters.

(path_driver_cb_baton_t,

svn_client__shelf_replay): Remove them from the path driver baton.

  1. … 2 more files in changeset.
Fix a regression in the svn_delta_path_driver2().

Introduced in r1851738.

* subversion/libsvn_delta/path_driver.c

(svn_delta_path_driver_step): Restore predictable behaviour for callbacks

that don't set the 'dir baton' output parameter.

Teach the delta editor path driver to work incrementally.

Instead of passing in the complete list of paths to be driven all at once,

this adds the option of passing in one path at a time.

* subversion/include/svn_delta.h,

subversion/libsvn_delta/path_driver.c

(svn_delta_path_driver2): Rewrite to use the incremental API.

(svn_delta_path_driver_state_t,

svn_delta_path_driver_start,

svn_delta_path_driver_step,

svn_delta_path_driver_finish): New.

  1. … 1 more file in changeset.
Fix an issue in the svn_txdelta_to_svndiff_stream() API that could cause

unexpected short reads (EOFs) on the stream.

This API is used when performing PUT requests to the server. Consequently,

the bug could result in truncated payload being sent to the server and

failing commits over http://.

* subversion/libsvn_delta/svndiff.c

(svndiff_stream_read_fn): Handle a case where we have received the final

window from the txdelta stream, but the remaining part of the buffer

cannot fully accommodate it during this call to read_fn. Instead of

exiting and triggering an unexpected short read, allow the remaining

part to be read during subsequent calls to read_fn.

* subversion/tests/libsvn_delta

(): Add 'svndiff-stream-test' to svn:ignore.

* subversion/tests/libsvn_delta/svndiff-stream-test.c: New file with a

regression test for this issue.

* build.conf

(svndiff-stream-test): New.

(__ALL_TESTS__): Run svndiff-stream-test.

  1. … 3 more files in changeset.
* subversion/libsvn_delta/debug_editor.c

(svn_delta__get_debug_editor): Use the proper constructor for the editor.

Let the 'debug editor' work even without a wrapped editor.

* subversion/libsvn_delta/debug_editor.c

Only call into the wrapped editor if it is not null.

Let the 'debug editor' support the new-in-1.10 'apply_txdelta_stream' method.

* subversion/libsvn_delta/debug_editor.c

(apply_textdelta_stream): New.

(svn_delta__get_debug_editor): Add the new method.

* subversion/include/private/svn_element.h,

subversion/libsvn_delta/element.c

(svn_element__tree_set): Remove unused 'svn_error_t *' return.

Found by: danielsh

using 'tools/dev/warn-unused-result.sh'

  1. … 1 more file in changeset.
* subversion/libsvn_delta/text_delta.c

(apply_window): Use the standard error handling pattern with SVN_ERR()

and tighten the scope of the local variable.

Minor code cleanup: use svn checksum routines instead of calling apr_md5

functions in the implementation of svn_txdelta_apply().

* subversion/libsvn_delta/text_delta.c

(struct apply_baton): Make md5_context an svn_checksum_ctx_t.

(apply_window): Replace apr_md5_final() with svn_checksum_final(),

compose and handle potential errors. Replace apr_md5_update() with

svn_checksum_update().

(svn_txdelta_apply): Update context initialization.

Following up on r1803140, comment on why the memory usage of the new

svn_txdelta_to_svndiff_stream() function is limited, and on the fact

that we rely on the implementation detail of svn_txdelta_to_svndiff3().

See the related discussion in:

https://lists.apache.org/thread.html/51c8418956457d2739e2672340fbcacc86d9ee40be7f8dae447e998d@%3Cdev.subversion.apache.org%3E

* subversion/libsvn_delta/svndiff.c

(window_handler, svndiff_stream_write_fn, svn_txdelta_to_svndiff_stream):

Add comments and cross-link them.

Lay the groundwork that would allow ra_serf to stream svndiff deltas

without creating temporary files when working against new servers.

This patch adds a new delta editor callback, apply_textdelta_stream()

that we'll use to stream the deltas, fully implements in the ra_serf's

commit editor, but doesn't yet change the behavior of existing editor

drivers, such as svn_client_import5() or svn_client_commit6().

This requires a minor tweak to the Subversion's HTTP protocol, and

it's the reason why streaming would only work against new servers.

Currently, all PUT requests include a special header that contains the

result checksum, which is used by the server to validate the integrity

of the result after it applies the delta received over the wire. While

this approach works fine if the client first creates a temporary file with

the delta and only then starts sending it to the server (the result checksum

is calculated while preparing the temporary file), it can't be used in the

stream approach, as with it we'd need to know the result checksum _before_

we start sending data.

So we turn the existing scheme inside out, and teach mod_dav_svn to send the

result checksum in the responses to PUT requests. Then, the client would

check if the received checksum matches what it calculated, and, possibly,

return a checksum mismatch error (thus aborting the edit and the transaction).

* subversion/include/svn_delta.h

(svn_txdelta_stream_open_func_t): New callback type.

(svn_delta_editor_t): Add a forward declaration for this type.

(svn_delta_editor_t.apply_textdelta_stream): New vtable member.

* subversion/libsvn_delta/default_editor.c

(apply_textdelta_stream): Provide default implementation for this callback

that performs a fallback to apply_textdelta().

(default_editor): Extend the default instance of an svn_delta_editor_t.

* subversion/libsvn_delta/cancel.c

(apply_textdelta_stream): Implement this forwarding callback, and ...

(svn_delta_get_cancellation_editor): ...install it here.

* subversion/include/svn_dav.h

(SVN_DAV_NS_DAV_SVN_PUT_RESULT_CHECKSUM): New.

* subversion/mod_dav_svn/repos.c

(close_stream): Send the "X-SVN-Result-Fulltext-MD5" header when responding

to successful PUT requests.

* subversion/mod_dav_svn/version.c

(get_vsn_options): Advertise new capability.

* subversion/libsvn_ra_serf/ra_serf.h

(svn_ra_serf__session_t.supports_put_result_checksum): New field.

* subversion/libsvn_ra_serf/options.c

(capabilities_headers_iterator_callback): Remember if the server sends

the result checksum in the response to a successful PUT request.

* subversion/libsvn_ra_serf/commit.c

(file_context_t.svndiff_sent): New field.

(file_context_t.remote_result_checksum): New field.

(apply_textdelta): Update the comment stating that it would be nice to

get rid of temporary files for svndiff delta. Factor out the svndiff

format selection logic ...

(negotiate_put_encoding): ...into this new helper function.

(open_txdelta_baton_t): New.

(txdelta_stream_errfunc): New error function for the stream bucket.

(create_body_from_txdelta_stream): New svn_ra_serf__request_body_delegate_t

that creates the request body by opening an svn_txdelta_stream_t, turning

it into svn_stream_t and giving away a bucket wrapping around that stream.

Use it in ...

(apply_textdelta_stream): ...this new function that performs a PUT, and

streams the request body.

(put_response_ctx_t): New.

(put_response_handler): New, remembers the result checksum we received

from the server.

(close_file): Don't do a PUT if we did it in apply_textdelta_stream().

Check for a checksum mismatch using the checksum returned from the server.

(svn_ra_serf__get_commit_editor): Install the new apply_textdelta_stream()

callback when working against new servers.

  1. … 7 more files in changeset.
Add new svn_txdelta_to_svndiff_stream() API.

This API turns a given svn_txdelta_stream_t delta stream into a readable

svn_stream_t containing svndiff. It would be required to make ra_serf

stream svndiff deltas without creating temporary files.

* subversion/include/svn_delta.h

(svn_txdelta_to_svndiff_stream): Declare new function.

* subversion/libsvn_delta/svndiff.c

(svndiff_stream_baton_t): New.

(svndiff_stream_write_fn): New, writes svndiff to a temporary buffer.

(svndiff_stream_read_fn): New, passes the buffered data to the caller and

refills the internal buffer if necessary.

(svn_txdelta_to_svndiff_stream): Implement this new function. Set up

pull- and push-streams. Install the read and write stream handlers, and

return the pull stream to the caller.

* subversion/tests/libsvn_delta/random-test.c

(do_random_txdelta_to_svndiff_stream_test): Core of the new test, similar

to do_random_test().

(random_txdelta_to_svndiff_stream_test): Run new test.

(test_funcs): Add new test.

  1. … 2 more files in changeset.
fsfs: Add initial support for LZ4 compression.

This can significantly (up to 3 times) improve the speed of commits and

other operations with large binary or incompressible files, while still

maintaining a decent compression ratio.

Our current use of zlib compression — which, depending on the protocol,

can be used multiple times — heavily affects the speed of commits with

large binary or incompressible files. According to the Squash benchmark

(https://quixdb.github.io/squash-benchmark/) and to my measurements, the

zlib compression speed with the default level is about 30-40 MiB/s, and

it doesn't matter if the file is incompressible or not.

This patch provides an alternative in the form of the LZ4 compression.

While still providing a decent compression ratio, LZ4 offers much faster

compression even than zlib with level=1, and can skip incompressible data

chunks. Presumably, LZ4 is used for on-the-fly compression in different

file systems for these reasons.

With this patch, LZ4 compression will be enabled for fsfs repositories which

specify compression-level=1 in fsfs.conf. The interoperability is implemented

by bumping the format of svndiff to 2 and the repository file system format

to 8. From the client perspective, the patch starts using LZ4 compression

only for file:// protocol, and the support/negotiation of the use of svndiff2

with LZ4 compression for http:// and svn:// can be added later.

The tests for LZ4 compression can be run with one of the following commands:

win-tests.py --fsfs-compression=1

make check FSFS_COMPRESSION=1

* subversion/include/svn_delta.h

(svn_txdelta_to_svndiff3): Update docstring.

* subversion/include/svn_error_codes.h

(SVN_ERR_LZ4_COMPRESSION_FAILED,

SVN_ERR_LZ4_DECOMPRESSION_FAILED): New error codes.

* subversion/include/private/svn_subr_private.h

(svn__compress, svn__decompress): Rename to ...

(svn__compress_zlib, svn__decompress_zlib): ..this.

(svn__compress_lz4, svn__decompress_lz4): Declare new functions.

* subversion/libsvn_subr/compress.c

(): Include LZ4 library header.

(svn__compress, svn__decompress): Rename to ...

(svn__compress_zlib, svn__decompress_zlib): ..this.

(svn__compress_lz4, svn__decompress_lz4): Implement new functions.

* subversion/libsvn_subr/packed_data.c

(write_stream_data, read_stream_data): Update usages of svn__compress()

and svn__decompress().

* subversion/libsvn_delta/svndiff.c

(SVNDIFF_V2): New.

(get_svndiff_header): Update to support svndiff2 headers.

(encode_window, decode_window, write_handler): Support svndiff2 with

LZ4 compression. Tweak the relevant comments.

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__FORMAT_NUMBER): Bump to 8.

(SVN_FS_FS__MIN_SVNDIFF2_FORMAT): New define.

* subversion/libsvn_fs_fs/fs_fs.c

(write_config): Tweak the compression-level option description.

(svn_fs_fs__create, svn_fs_fs__info_format): Update to handle the

format bump.

* subversion/libsvn_fs_fs/transaction.c

(txdelta_to_svndiff): New helper to call svn_txdelta_to_svndiff3() with

appropriate svndiff version and compression level, depending on the

file system configuration.

(rep_write_get_baton, write_container_delta_rep): Use new helper.

* subversion/libsvn_fs_fs/revprops.c

(parse_packed_revprops, repack_revprops, svn_fs_fs__copy_revprops):

Update usages of svn__compress() and svn__decompress().

* subversion/libsvn_fs_fs/structure

(Filesystem formats): Update to describe usage of svndiff2.

* subversion/tests/libsvn_subr/compress-test.c: New.

* subversion/tests/libsvn_delta/random_test.c

(DEFAULT_ITERATIONS): Increase to 60.

(do_random_test, do_random_combine_test): Test different svndiff versions

and compresssion levels.

* build.conf

(libsvn_subr): Build LZ4 library sources.

(compress-test): Add new section.

* notes/svndiff: Describe svndiff2.

* NOTICE, LICENSE: Include license for LZ4.

  1. … 16 more files in changeset.
In 'svnmover', consolidate some branching methods: merge the 'branch' and

'open_branch' methods into one.

* subversion/include/private/svn_branch.h

(svn_branch__txn_open_branch): Add a 'tree_ref' parameter.

(svn_branch__txn_branch): Delete.

* subversion/include/private/svn_branch_impl.h

(svn_branch__txn_v_open_branch_t): Add a 'tree_ref' parameter.

(svn_branch__txn_v_branch_t): Delete.

(svn_branch__txn_vtable_t): Remove that method from the vtable.

* subversion/libsvn_delta/branch.c

(branch_txn_open_branch): Add a 'tree_ref' parameter, and apply it like

the 'branch' method used to do.

(branch_txn_branch): Delete.

(svn_branch__txn_open_branch): Add a 'tree_ref' parameter.

(svn_branch__txn_branch): Delete.

(branch_txn_create): Remove that method from the vtable.

* subversion/libsvn_delta/branch_compat.c

(compat_branch_txn_open_branch): Add a 'tree_ref' parameter.

(compat_branch_txn_branch): Delete.

(svn_branch__compat_txn_from_delta_for_commit): Remove that method from

the vtable.

* subversion/libsvn_delta/branch_nested.c

(svn_branch__instantiate_elements_r): Update caller: pass null.

(nested_branch_txn_open_branch): Add a 'tree_ref' parameter, and apply

recursion to it like nested_branch_txn_branch() used to do.

(nested_branch_txn_branch): Delete.

(svn_branch__nested_txn_create): Remove that method from the vtable.

* tools/dev/svnmover/merge3.c

(merge_subbranch): Use 'open_branch' instead of 'branch' method.

* tools/dev/svnmover/svnmover.c

(svn_branch__replay,

update_wc_base_r,

do_mkbranch): Update caller: pass null.

(do_branch,

do_topbranch): Use 'open_branch' instead of 'branch' method.

  1. … 4 more files in changeset.
In 'svnmover', consolidate some branching methods. Basically, delete the

two 'add branch' methods because 'open branch' can do the same job.

* subversion/include/private/svn_branch.h

(svn_branch__txn_get_branches): Add comments.

(svn_branch__txn_open_branch): Add comments. Rename a parameter.

(svn_branch__state_t): Add comments.

(svn_branch__txn_add_branch,

svn_branch__txn_add_new_branch): Delete.

* subversion/include/private/svn_branch_impl.h

(svn_branch__txn_v_add_branch_t,

svn_branch__txn_v_add_new_branch_t): Delete.

(svn_branch__txn_vtable_t): Delete those entries from the vtable.

* subversion/libsvn_delta/branch.c

(branch_txn_add_branch,

branch_txn_add_new_branch): Delete.

(branch_txn_open_branch,

branch_txn_branch): Don't use svn_branch__txn_add_new_branch().

(svn_branch__txn_add_branch,

svn_branch__txn_add_new_branch): Delete.

(svn_branch__txn_open_branch): Rename a parameter.

(branch_txn_create): Update the vtable.

(svn_branch__txn_parse): Don't use svn_branch__txn_add_branch().

* subversion/libsvn_delta/branch_compat.c

(compat_branch_txn_add_branch,

compat_branch_txn_add_new_branch): Delete.

(svn_branch__compat_txn_from_delta_for_commit): Update the vtable.

* subversion/libsvn_delta/branch_nested.c

(svn_branch__instantiate_elements_r): Use svn_branch__txn_open_branch()

instead of ...add_new_branch.

(nested_branch_txn_add_branch,

nested_branch_txn_add_new_branch): Delete.

(svn_branch__nested_txn_create): Update the vtable.

* tools/dev/svnmover/svnmover.c

(do_mkbranch): Add a comment.

  1. … 3 more files in changeset.
Remove the debug code inserted in r1719056, r1719067, r1719072, r1719074.

* subversion/libsvn_delta/branch_compat.c

(pathrev_str,

hash_keys_str,

change_node_str): Comment out, as now unused but might be useful again.

(insert_change,

drive_changes_r): Remove recent (and some older) debugging code.

In 'svnmover', track history better.

For each version of each branch, store pointers to previous branch-versions,

called "parents" like in Git. Usually, a branch version has one parent that

points to the previous version of the same branch. Upon branching, the new

branch gets one parent that points to the chosen version of the source

branch. Upon merging, in the usual case of a complete merge from a source

branch into a target branch, the target branch gets two parents: its own

previous version, and the merge source.

* subversion/include/private/svn_branch.h

(svn_branch__txn_open_branch,

svn_branch__txn_add_new_branch): Don't take a 'predecessor' parameter.

(svn_branch__state_t): Don't track a 'predecessor' as a public field here.

(svn_branch__history_t,

svn_branch__history_create_empty,

svn_branch__history_create,

svn_branch__history_dup): New.

(svn_branch__state_get_history): Rename from

'svn_branch__state_get_merge_ancestor'.

(svn_branch__state_set_history): Rename from

'svn_branch__state_add_merge_ancestor'.

* subversion/include/private/svn_branch_impl.h

Track the changes in svn_branch.h.

* subversion/libsvn_delta/branch.c

Implement the changes in svn_branch.h.

* subversion/libsvn_delta/branch_compat.c

Track the changes in svn_branch.h.

* subversion/libsvn_delta/branch_nested.c

Track the changes in svn_branch.h.

* subversion/tests/cmdline/svnmover_tests.py

(reported_mg_diff): No longer expect any diff output describing a merge

history difference, as we don't print that in a diff any more.

* tools/dev/svnmover/ra.c

(branch_get_mutable_state): Initialize each branch's parent to point to

the same branch in the base revision.

* tools/dev/svnmover/svnmover.c

(list_parents,

history_str,

svn_branch__history_add_parent): New.

(history_diff): Rename from 'merge_history_diff'.

(txn_is_changed): Don't compare history, but leave some place-holder code

in case we want to do so later.

(get_union_of_subbranches): Make robust against either or both inputs

being null.

(svn_branch__replay): Update the replaying of a change of history.

(update_wc_base_r): Update the copying of history for a new branch.

(do_merge): Update the recording of history.

(do_auto_merge): Dummy implementation: does nothing.

(show_history_r): New.

(branch_diff_r): Don't show history differences.

(find_branch_main_parent): New.

(svn_branch__find_predecessor_el_rev): Update the finding of the main

parent.

(do_log): Also show the history before the diff of each revision.

(do_mkbranch): Track the API changes.

(do_branch): Initialize the history (parent) of the new branch.

(show_branch_history): New.

(execute): In the 'info-wc' subcommand, show the history of the working

branch (and not of the base branch).

  1. … 5 more files in changeset.
Continue debugging: try avoiding apr_hash_overlay() as it appears to be

returning a hash with duplicate keys in it (which should be impossible),

on the Mac buildbots. A follow-up to r1719056.

* subversion/libsvn_delta/branch_compat.c

(hash_overlay): New.

(drive_changes_r): Use hash_overlay() instead of apr_hash_overlay().

Change the debug code to return the info as an error message text, as that

may be logged by the buildbots whereas stdout/stderr are not. A follow-up

to r1719056.

* subversion/libsvn_delta/branch_compat.c

(insert_change,

drive_changes_r): Report debugging info through an error object.

Enable the debug code also in release builds, as the buildbots are doing

release builds. A follow-up to r1719067.

* subversion/libsvn_delta/branch_compat.c

(change_node_str): Don't make debugging info conditional on SVN_DEBUG.

Enable the debug code also in release builds, as the buildbots are doing

release builds. A follow-up to r1719056.

[Note from the future: Corrected in r1719072.]

* subversion/libsvn_delta/branch_compat.c

(pathrev_str,

hash_keys_str,

insert_change,

drive_changes_r): Don't make debugging info conditional on SVN_DEBUG.

Add some debugging in 'svnmover' compatibility code to try to diagnose

failures that show up on some buildbots but not on my system.

* subversion/libsvn_delta/branch_compat.c

(pathrev_str,

hash_keys_str,

change_node_str): New.

(insert_change,

drive_changes_r): Print debugging info if there's an assertion failure.

* subversion/libsvn_delta/element.c

(svn_eid__hash_sorted_first,

svn_eid__hash_sorted_next): Tweak a cast to avoid a 'discarding const'

warning that GCC can produce (even though the warning was poor

because the code only used the pointer for reading).

* subversion/libsvn_delta/element.c

(svn_element__payload_invariants): Tweak conditionals to avoid

compiler warning.

Revert a tweak commited accidentally in r1717957.

* subversion/libsvn_delta/branch.c

(svn_branch__txn_get_branch_by_id): Remove the extra 'return'.