Checkout Tools
  • last updated 3 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
In FSFS, update all remaining calling sites so that they would respect the

new filesystem config option, SVN_FS_CONFIG_NO_FLUSH_TO_DISK.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__write_min_unpacked_rev): Don't flush to disk if that's

allowed.

* subversion/libsvn_fs_fs/fs_fs.c

(svn_fs_fs__write_format, svn_fs_fs__set_uuid): Don't flush to disk

if that's allowed.

* subversion/libsvn_fs_fs/revprops.h

(svn_fs_fs__copy_revprops, svn_fs_fs__pack_revprops_shard): Accept new

`flush_to_disk' argument.

* subversion/libsvn_fs_fs/revprops.c

(svn_fs_fs__copy_revprops, svn_fs_fs__pack_revprops_shard): Make the

flush optional based on the new argument.

(svn_fs_fs__upgrade_pack_revprops, write_non_packed_revprop,

switch_to_new_revprop, repack_revprops, write_packed_revprop): Don't

flush to disk if that's allowed.

* subversion/libsvn_fs_fs/pack.c

(pack_context_t): Add `flush_to_disk' boolean field.

(initialize_pack_context): Accept new `flush_to_disk' argument and set

the new field in pack_context_t accordingly.

(close_pack_context): Make the flush optional based on the stored state

in the pack context.

(pack_rev_shard): Accept new `flush_to_disk' argument, propagate it ...

(pack_log_addressed): ...here...

(pack_phys_addressed): ...and here.

(pack_shard): Adjust call to pack_rev_shard() and don't flush to disk

if that's allowed.

(synced_pack_shard): Adjust call to svn_fs_fs__pack_revprops_shard().

  1. … 4 more files in changeset.
Introduce `--no-flush-to-disk' option for `svnadmin load'.

The option can be used to to dramatically speed up the load process when

there's no need to ensure that the resulting data survives a system crash

or power loss — e.g., when loading a dump into a fresh new repository.

This is one of the ideas in http://svn.haxx.se/dev/archive-2015-09/0187.shtml

(Subject: "Whiteboard -- topics list on the white board").

* subversion/include/svn_fs.h

(SVN_FS_CONFIG_NO_FLUSH_TO_DISK): New option.

* subversion/libsvn_fs_fs/fs.h

(fs_fs_data_t): Add `flush_to_disk' boolean field.

* subversion/libsvn_fs_fs/fs.с

(initialize_fs_struct): Initialize the new field.

* subversion/libsvn_fs_fs/fs_fs.c

(read_global_config): Set the new field based on what's in fs->config.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__move_into_place): Accept a new `flush_to_disk' argument.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__move_into_place): Make the flush optional based on the

new argument.

* subversion/libsvn_fs_fs/transaction.c

(get_and_increment_txn_key_body): Don't flush to disk if that's allowed.

(write_final_revprop): Accept a new `flush_to_disk' argument. Make the

flush optional based on the new argument.

(commit_body): Don't flush to disk if that's allowed. Adjust calls to

write_final_revprop() and svn_fs_fs__move_into_place().

* subversion/libsvn_fs_fs/revprops.c

(switch_to_new_revprop): Adjust the call to svn_fs_fs__move_into_place().

Keep the existing behavior and always flush to disk.

* subversion/svnadmin/svnadmin.c

(svnadmin__no_flush_to_disk): New enum value.

(options_table): Define --no-flush-to-disk option.

(cmd_table): Allow `load' to accept --no-flush-to-disk.

(svnadmin_opt_state): Add `no_flush_to_disk' member.

(open_repos): Move below the definition of svnadmin_opt_state. Accept

an svnadmin_opt_state structure as one of the arguments and initialize

the new SVN_FS_CONFIG_NO_FLUSH_TO_DISK option based on it.

(subcommand_crashtest, subcommand_deltify, subcommand_dump,

subcommand_dump_revprops, subcommand_load, subcommand_load_revprops,

subcommand_lstxns, subcommand_recover, subcommand_rmtxns, set_revprop,

subcommand_setuuid, subcommand_pack, subcommand_verify, subcommand_info,

subcommand_lock, subcommand_lslocks, subcommand_rmlocks,

subcommand_unlock): Adjust these callers of open_repos().

(main): Handle --no-flush-to-disk option.

* subversion/tests/cmdline/svnadmin_tests.py

(load_no_flush_to_disk): New test.

(test_list): Add reference to new test.

* tools/client-side/bash_completion

(_svnadmin): Add new option to `load'.

  1. … 10 more files in changeset.
Promote libsvn_fs_fs private helper svn_fs_fs__get_file_offset() to public

libsvn_subr function svn_io_file_get_offset().

* subversion/libsvn_fs_fs/util.c

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__get_file_offset): Move/rename to ...

* subversion/include/svn_io.h

* subversion/libsvn_subr/io.c

(svn_io_file_get_offset): ... here.

* subversion/libsvn_fs_fs/cached_data.c

* subversion/libsvn_fs_fs/index.c

* subversion/libsvn_fs_fs/pack.c

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/verify.c

(get_file_offset, stream_error_create, copy_item_to_temp, copy_rep_to_temp,

copy_node_to_temp, rep_write_get_baton, rep_write_contents_close,

write_container_rep, write_container_delta_rep, write_final_rev,

write_final_changed_path_info, commit_body, expect_buffer_nul): Replace

calls to svn_fs_fs__get_file_offset() within svn_io_file_get_offset().

  1. … 8 more files in changeset.
Implement svn_io_write_atomic2() with FLUSH_TO_DISK flag to control whether

wait or not until file is actually written to disk. The old

svn_io_write_atomic() was flushing data to disk unconditionally.

* subversion/include/svn_io.h

(svn_io_write_atomic2): New function declaration.

(svn_io_write_atomic): Deprecate.

* subversion/libsvn_subr/io.c

(svn_io_write_atomic2): Revv from svn_io_write_atomic() Add FLUSH_TO_DISK

parameter and perform flush to disk only if FLUSH_TO_DISK is non-zero.

* subversion/libsvn_subr/deprecated.c

(svn_io_write_atomic): Call svn_io_write_atomic2() with FLUSH_TO_DISK=TRUE.

* subversion/libsvn_fs_fs/fs_fs.c

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/util.c

* subversion/libsvn_fs_x/fs_x.c

* subversion/libsvn_fs_x/revprops.c

* subversion/libsvn_fs_x/util.c

* subversion/libsvn_wc/workqueue.c

* subversion/mod_dav_svn/activity.c

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

* subversion/tests/libsvn_repos/repos-test.c

(*): Use svn_io_write_atomic2() with FLUSH_TO_DISK=TRUE instead of

svn_io_write_atomic().

  1. … 13 more files in changeset.
Use the native Windows API capabilities to guarantee that data is flushed

to disk after file renames in FSFS. This also avoids reopening the renamed

file just to perform the flush.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__move_into_place): Call svn_io_file_rename2() with

FLUSH_TO_DISK=TRUE and remove flushing to disk after rename, since

svn_io_file_rename2() will do this for us. We still have to manually

flush changes for cross-volume copies, but it's something that could

be improved later.

Implement svn_io_file_rename2() with FLUSH_TO_DISK flag to require OS to

wait until rename operation is actually written to disk. Discussed in thread

"svn commit: r1682265 - /subversion/trunk/subversion/libsvn_fs_fs/util.c" on

dev@s.a.o [1]

[1] http://svn.haxx.se/dev/archive-2015-05/0211.shtml

* subversion/include/svn_io.h

(svn_io_file_rename2): New function declaration.

(svn_io_file_rename): Deprecate.

* subversion/libsvn_subr/io.c

(win32_file_rename): Use MOVEFILE_WRITE_THROUGH flag in call to

MoveFileExW if FLUSH_TO_DISK is non-zero.

(svn_io_file_rename2): Revv from svn_io_file_rename(). Add FLUSH_TO_DISK

parameter and perform flush to disk operation depending on the platform:

use MoveFileEx flag on Windows, fsync() target directory on POSIX and

fsync() target file on all other platforms. This logic mostly copied

from svn_fs_fs__move_into_place().

(svn_io_copy_link, svn_io_copy_file, svn_io_write_atomic,

svn_io_write_version_file): Use svn_io_rename2() with

FLUSH_TO_DISK=FALSE instead of svn_io_rename().

* subversion/tests/libsvn_subr/io-test.c

(test_file_rename2): Simple tests for svn_io_file_rename2().

(test_funcs): Add test_file_rename2.

* subversion/libsvn_subr/deprecated.c

(svn_io_file_rename): Call svn_io_file_rename2() with FLUSH_TO_DISK=FALSE.

* subversion/libsvn_client/copy.c

* subversion/libsvn_client/export.c

* subversion/libsvn_client/externals.c

* subversion/libsvn_fs_fs/fs_fs.c

* subversion/libsvn_fs_fs/lock.c

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/util.c

* subversion/libsvn_fs_x/lock.c

* subversion/libsvn_fs_x/transaction.c

* subversion/libsvn_fs_x/util.c

* subversion/libsvn_subr/config_auth.c

* subversion/libsvn_subr/stream.c

* subversion/libsvn_subr/subst.c

* subversion/libsvn_wc/copy.c

* subversion/libsvn_wc/node.c

* subversion/libsvn_wc/upgrade.c

* subversion/libsvn_wc/wc_db_pristine.c

* subversion/libsvn_wc/workqueue.c

(*): Use svn_io_file_rename2() with FLUSH_TO_DISK=FALSE instead of

svn_io_file_rename().

  1. … 21 more files in changeset.
Where feasible, we want use simple definedness tests with our SVN_*

preprocessor flag macros. So, make SVN_ON_POSIX comply to this policy.

* subversion/include/private/svn_dep_compat.h

(SVN_ON_POSIX): Switch from a value-based preprocessor flag to simple

definedness.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__move_into_place): Update SVN_ON_POSIX check.

* subversion/libsvn_fs_x/batch_fsync.c

(svn_fs_x__batch_fsync_new_path): Same.

* subversion/libsvn_subr/io.c

(svn_io_write_atomic): Same.

Suggested by: brane

  1. … 3 more files in changeset.
Correctly fsync() after renames in FSFS on Win32. We must flush the disk

buffers after the rename, otherwise the metadata may not be persistent.

Moreover, if the rename is degraded to a copy by Win32, we won't even have

the complete file contents on disk without a buffer flush.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__move_into_place): On Win32, always flush after rename.

Fix our usage of fsync usage on non-Linux POSIX platforms.

They all share the "directory contains the file name" property.

* subversion/include/private/svn_dep_compat.h

(SVN_ON_POSIX): New define.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__move_into_place): Always sync the parent directory when

we are on POSIX - not just for Linux.

* subversion/libsvn_subr/io.c

(svn_io_write_atomic): Same.

  1. … 2 more files in changeset.
Follow-up to r1637184: Remove some unused code.

* subversion/libsvn_fs_fs/fs_fs.c

(upgrade_body): Remove unused TXNS_DIR local variable and it's

initialization.

* subversion/libsvn_fs_fs/transaction.c

(is_noderev_file, fnv1a_checksum_on_file_range): Remove unused functions.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_txns_dir): Remove unused variable.

  1. … 2 more files in changeset.
Make FSFSv7 repositories always use consistent addressing mode, instead of

saving revision number from which logical addressing was enabled.

From the performance point of view there will be no big benefits to enable

log addressing for an existing repository, because the existing old part

of the repository will remain to be addressed physically. So those who

want to benefit from the performance improvements related to the log

addressing feature will be required to dump/load their repositories anyway.

On the other hand, consistent addressing allows us to omit some tricky code.

This also fixes problems with long living svn_fs_t instances during the hot

repository upgrade in background.

* subversion/libsvn_fs_fs/fs.h

(PATH_TXNS_LA_DIR): Remove.

(fs_fs_data_t): Replace MIN_LOG_ADDRESSING_REV member with simple

USE_LOG_ADDRESSING flag.

* subversion/libsvn_fs_fs/fs.c

(initialize_fs_struct): Initialize USE_LOG_ADDRESSING field instead of

MIN_LOG_ADDRESSING_REV.

* subversion/libsvn_fs_fs/util.c

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__use_log_addressing): Remove REV argument.

* subversion/libsvn_fs_fs/cached_data.c

(dbg_log_access, use_block_read, svn_fs_fs__rev_get_root,

svn_fs_fs__check_rep, svn_fs_fs__get_changes): Adapt calls to

svn_fs_fs__use_log_addressing().

* subversion/libsvn_fs_fs/dump-index.c

(svn_fs_fs__dump_index): Adapt calls to svn_fs_fs__use_log_addressing().

* subversion/libsvn_fs_fs/fs_fs.c

(read_format): Replace MIN_LOG_ADDRESSING_REV output parameter with

USE_LOG_ADDRESSING. Do not parse revision number if repository uses

logical addressing mode.

(svn_fs_fs__write_format): Simple write "addressing physical" or

"addressing logical" to format file depending repository addressing mode.

(svn_fs_fs__read_format_file): Adapt calls to read_format() function.

(upgrade_body): Replace local variable MIN_LOG_ADDRESSING_REV with

USE_LOG_ADDRESSING flag. Adapt calls to read_format() and remove

code that renames transaction folder during upgrade.

(write_revision_zero): Adapt calls to svn_fs_fs__use_log_addressing().

(svn_fs_fs__create_file_tree): Replace MIN_LOG_ADDRESSING_REV argument with

USE_LOG_ADDRESSING flag and use it for FFD->USE_LOG_ADDRESSING

initialization.

(svn_fs_fs__create): Pass TRUE for USE_LOG_ADDRESSING parameter in call to

svn_fs_fs__create_file_tree().

* subversion/libsvn_fs_fs/fs_fs.h

(svn_fs_fs__create_file_tree): Update function declaration and docstring.

* subversion/libsvn_fs_fs/hotcopy.c

(hotcopy_create_empty_dest): Adapt call to svn_fs_fs__create_file_tree().

* subversion/libsvn_fs_fs/load-index.c

* subversion/libsvn_fs_fs/index.c

(svn_fs_fs__item_offset, svn_fs_fs__load_index): Adapt calls

to svn_fs_fs__use_log_addressing().

* subversion/libsvn_fs_fs/pack.c

(svn_fs_fs__order_dir_entries, pack_rev_shard): Adapt calls

to svn_fs_fs__use_log_addressing().

* subversion/libsvn_fs_fs/stats.c

(parse_representation, read_noderev, read_revisions): Adapt calls

to svn_fs_fs__use_log_addressing().

* subversion/libsvn_fs_fs/transaction.c

(auto_truncate_proto_rev, store_p2l_index_entry, allocate_item_index,

write_final_rev): Adapt calls to svn_fs_fs__use_log_addressing().

(store_l2p_index_entry): Remove FINAL_REVISION argument. Adapt calls

to svn_fs_fs__use_log_addressing().

(compare_sort_p2l_entry, using_log_addressing, upgrade_transaction):

Remove.

(commit_body): Remove call to upgrade_transaction().

* subversion/libsvn_fs_fs/verify.c

(svn_fs_fs__verify): Adapt calls to svn_fs_fs__use_log_addressing().

* subversion/tests/libsvn_fs/fs-test.c

(upgrade_while_committing): Expect that begin_txn and commit doesn't fail

after upgrade to fsfs7.

  1. … 15 more files in changeset.
Follow-up to r1636478: Style change. No functional change.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_txns_dir): Put the parantheses around the whole

conditional expression.

Suggested by: brane

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_txns_dir): Add parantheses around condition.

No function change.

Suggested by: brane

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__read_min_unpacked_rev): Use a verifying rev num parser.

Make FSFS format 7 use a different name for the 'transactions' folder.

This is somewhat similar to what happened during the f1/f2->f3 upgrade

which made old code either succeed with a commit or fail in various

places when the format upgrade already had an impact on the contents.

With this change, however, old servers will not be able to even access

txns after the upgrade and will consistently see ENOENT errors. Hence,

they can't write to the repo. New servers will only be able to continue

transactions after a hot upgrade when they refresh their format info

as done in e.g. r1627949.

* subversion/libsvn_fs_fs/fs.h

(PATH_TXNS_DIR): Document that this is only valid for pre-f7 repos.

(PATH_TXNS_LA_DIR): New path constant.

* subversion/libsvn_fs_fs/fs_fs.c

(upgrade_body): Rename the txns dir if the repo has been pre-f7.

Be sure to revert that if the upgrade process failed.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_txns_dir): Return the now format-dependent txns folder.

* subversion/libsvn_fs_fs/structure

(Layout of the FS directory): Document how the txns folder name depends

on the repo format.

* subversion/tests/libsvn_fs/fs-test.c

(upgrade_while_committing): Adapt test case. Any txn access with an

outdated svn_fs_t will now fail with ENOENT.

  1. … 4 more files in changeset.
Introduce path construction utilities for the 'transactions' and

'txn-protorevs' directory in FSFS repositories.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__path_txns_dir,

svn_fs_fs__path_txn_proto_revs): Declare new private API.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_txns_dir): Implement new private API.

(svn_fs_fs__path_txn_dir): Use the new function.

(svn_fs_fs__path_txn_proto_revs): Implement new private API.

(svn_fs_fs__path_txn_proto_rev,

svn_fs_fs__path_txn_proto_rev_lock): Use the new function.

* subversion/libsvn_fs_fs/fs_fs.c

(upgrade_body,

svn_fs_fs__create_file_tree): Use the new path constructors.

* subversion/libsvn_fs_fs/transaction.c

(create_txn_dir_pre_1_5,

svn_fs_fs__list_transactions): Same.

  1. … 3 more files in changeset.
Fix a broken assumption within the r1603605 changeset.

When format 1 and 2 filesystems are being upgraded, the upgrade routine

leaves the db/current contents as is. As a consequence, there is a window

when a filesystem has a new format, but the 'current' file still contains the

additional IDs left from an old format, i.e. it could look like "359 j5 v\n"

(last values are bogus and won't be there after the next transaction commit).

We should be able to parse it, otherwise any upgraded pre-1.4-compatible

repository will block the commits.

Loosen the corresponding checks.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__read_current): Do not be too strict for new filesystem formats

and only expect a parseable revision number. Adjust a scope of the local

variable.

* subversion/tests/cmdline/svnadmin_tests.py

(upgrade): New regression test. Prior to this change, we actually had no

tests covering the 'svnadmin upgrade' behavior, so we use this opportunity

to test both BDB and FSFS. Even though BDB is deprecated, its upgrade code

does some amount of work in this case, and there should be nothing wrong

with testing it.

(test_list): Reference the new test.

  1. … 1 more file in changeset.
Minor FSFS refactoring: move a common code sequence to util.*

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__packed_base_rev): Declare new internal API.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__packed_base_rev): Implement it with code taken from rev_file.c.

* subversion/libsvn_fs_fs/rev_file.c

(init_revision_file,

open_pack_or_rev_file): Call the new function.

* subversion/libsvn_fs_fs/verify.c

(packed_base_rev): Remove local function.

(verify_index_consistency): Call the new one.

  1. … 3 more files in changeset.
Append index data to the rev data file instead of using separate files.

This gives unpacked FSFS format 7 similar I/O characteristics and disk

space usage as earlier formats. It also eliminates the need for retries

after a potential background pack run because each file is now always

consistent with itself (earlier, data or index files might already have

been deleted while the respective other was still valid). Overall,

most of this patch is removing code necessary to handle separate files.

The new file format is simple:

<rep data><l2p index><p2l index><footer><footer length>

with the first three being identical what we had before. <footer length>

is a single byte giving the length of the preceding footer, so it's

easier to extract than the pre-f7 rev trailer and there is only one

per pack / rev file.

The footer itself is just two decimal file offsets marking the start

of the index sections, separated by a single space char.

* subversion/libsvn_fs_fs/low_level.h

(svn_fs_fs__parse_footer,

svn_fs_fs__unparse_footer): Declare new parser / generator API for the

f7 footer, similar to f6- revision trailer

but slightly simplified.

* subversion/libsvn_fs_fs/low_level.c

(svn_fs_fs__parse_footer,

svn_fs_fs__unparse_footer): Implement.

* subversion/libsvn_fs_fs/rev_file.h

(svn_fs_fs__revision_file_t): Add members for footer contents access

block size.

(svn_fs_fs__reopen_revision_file): Drop. Retry is no longer necessary.

(svn_fs_fs__auto_read_footer): Declare new function that reads the footer

if we didn't yet.

* subversion/libsvn_fs_fs/rev_file.c

(svn_fs_fs__reopen_revision_file): Drop.

(svn_fs_fs__auto_read_footer): Implement.

(svn_fs_fs__close_revision_file): Index streams don't carry any releasable

resources anymore.

* subversion/libsvn_fs_fs/index.h

(svn_fs_fs__packed_stream_close): Drop. There are no independent file

resources linked to the index data

streams anymore.

(svn_fs_fs__l2p_index_create,

svn_fs_fs__p2l_index_create): Rename to ...

(svn_fs_fs__l2p_index_append

svn_fs_fs__p2l_index_create): ... these and accept an open file instead

of a file name to be created.

* subversion/libsvn_fs_fs/index.c

(packed_stream_open): Instead of opening our own index file, accept an

already open one and use the specified offset range.

(svn_fs_fs__packed_stream_close): Drop.

(index_create): Drop. The files already exist.

(svn_fs_fs__l2p_index_create,

svn_fs_fs__p2l_index_create): Rename to ...

(svn_fs_fs__l2p_index_append

svn_fs_fs__p2l_index_create): ... these and use the already open file

instead of creating new ones.

(retry_open_l2p_index,

retry_open_p2l_index): No longer necessary.

(svn_fs_fs__item_offset,

svn_fs_fs__p2l_get_max_offset): No lookup retry necessary anymore.

(auto_open_l2p_index,

auto_open_p2l_index): Make sure we know the index location and then

open the stream based on the existing file.

* subversion/libsvn_fs_fs/transaction.h

(svn_fs_fs__add_index_data): New private API that combines index data

with the rep data file and finalizes it.

* subversion/libsvn_fs_fs/transaction.c

(svn_fs_fs__add_index_data): Implement.

(commit_body): Use the above.

* subversion/libsvn_fs_fs/fs_fs.c

(write_revision_zero): Update r0 template by simply appending the

index contents and a final footer.

* subversion/libsvn_fs_fs/cached_data.c

(block_read): Remove the now unnecessary retry-on-pack code.

* subversion/libsvn_fs_fs/hotcopy.c

(hotcopy_copy_shard_file): Remove index file copy code.

(hotcopy_copy_packed_shard,

hotcopy_revisions): Update callers.

* subversion/libsvn_fs_fs/pack.c

(close_pack_context): Use the new transaction.h API to finalize the

index data.

(pack_range): Instead of ending with the file, revision content now ends

where L2P index data starts.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__path_l2p_index,

svn_fs_fs__path_p2l_index): Drop. These are no longer separate files.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_l2p_index,

svn_fs_fs__path_p2l_index): Drop.

* subversion/libsvn_fs_fs/verify.c

(compare_p2l_to_rev): Rev contents now ends where the L2P index begins.

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

(pack_filesystem): There are no separate index files anymore.

* subversion/tests/cmdline/svnadmin_tests.py

(read_l2p): Use contents of the rev file instead of a separate index file.

(set_changed_path_list): Parse footer and splice data correctly retaining

the index contents as-is.

* tools/server-side/svnfsfs/load-index-cmd.c

(write_p2l_index,

write_l2p_index): Instead of creating the final index files, create

the proto-index files only.

(load_index): Trim the old index from the pack / rev file and append

the new ones using our new transaction.h API.

  1. … 17 more files in changeset.
Following up on r1603485, make the 'db/current' parsing within the

svn_fs_fs__read_current() routine stricter.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__read_current): Expect a newline after the revision number

for new FS formats and after the next-copy-id for old formats. A well-

formed 'db/current' file always includes a trailing newline. Raise an

error when it is absent.

Do not corrupt the destination 'db/current' contents when hotcopying old FSFS

filesystems. Currently we use the "hotcopy-and-recover" approach for that.

When we need to finalize the operation by writing appropriate values into the

'db/current' file in the destination, we only know the revision number, but

for old filesystems we also have to know the (next-node-id, next-copy-id)

values. They exist in the source 'db/current' file, but for some reasons

we currently do not read them and only read the youngest revision number.

Instead of doing that, we attempt to *recover* them from the destination

and this approach is broken:

* Proper next-node-id and next-copy-id recovery requires a full

revision-by-revision scan, otherwise there is no guarantee that you are

not going to miss something. This is essentially what happens now: we

attempt to recover the ids based solely on the youngest revision and

sometimes miss the highest ones. As a consequence, the content of the

destination 'db/current' (which should point to the *unused* ids) will

clash with something that is already committed to the repository.

* You do not want to run a full recovery scan in order for the backup (!)

to work, especially for the --incremental case. Even more, depending on

the recovery code in the hotcopy implementation seems quite illogical.

Fix this by atomically reading the (youngest revision, next-node-id,

next-copy-id) tuple from the source and writing it to destination when

everything is done. We do not do any kind of checkpointing for old FSFS

filesystems, so it is just a read-once and write-once approach.

See http://svn.haxx.se/dev/archive-2014-06/0053.shtml

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__read_current): New function.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__read_current): New function.

* subversion/libsvn_fs_fs/hotcopy.c

(hotcopy_body): Use the new svn_fs_fs__read_current() helper for the source

filesystem. Split the revision copying logic for old and new formats.

Whenever we encounter a new format filesystem, just stick to the existing

approach. For old (1 or 2) formats, use the new hotcopy_revisions_old()

shortcut and do not think about the necessity to copy packed files or to

checkpoint the results. The revision copying logic for new formats now

resides ...

(hotcopy_revisions): ...in this new function.

(hotcopy_revisions_old): New function, containing the revision copying

logic for old formats.

(hotcopy_update_current): Only handle the new filesystem formats and ...

(hotcopy_update_current_old): ...move the support for old formats in this

new function.

* subversion/tests/cmdline/svnadmin_tests.py

(fsfs_hotcopy_old_with_id_changes): Remove XFail marker.

  1. … 3 more files in changeset.
Reverting experimental move-related APIs.

For details see http://svn.haxx.se/dev/archive-2014-05/0155.shtml

This effectively reverts revisions 1595600,1572044,1547039,1535341,

1535330,1535080,1535029,1534280,1532200,1527217,1527088,1526057,

1526014,1526013,1525484,1525467,1525464,1525463,1525462,1525460,

1525459,1525453,1525448,1525442,1525432,1525429 and 1525419.

Remove respective entries from CHANGES.

  1. … 43 more files in changeset.
Minor code consistency fix in FSFS: get rid of the redundant path_lock()

function and make the path to the pack lock file available through a

similar private API function.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__path_pack_lock): Declare this path constructor similarly to

the already existing one for the write lock.

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_pack_lock): Moved here from fs_fs.c:path_pack_lock().

* subversion/libsvn_fs_fs/fs_fs.c

(path_lock,

path_pack_lock): Drop as redundant.

(init_lock_baton,

svn_fs_fs__create): Call the functions provide from utils.h

instead of the static ones.

  1. … 2 more files in changeset.
Merge FSFS format 7 into /trunk, i.e. everything from the

fsfs-improvements branch. There were no conflicts.

For easier review, the following is a summary of the changes.

Please refer to either the in-source comments or the original

commit messages for details.

BRANCH-README contains a feature overview. The structure and

structure-indexes files contain a detailed description of the

new addressing concepts and their external representation.

* BRANCH-README

(): Replace the old description with that for FSFS f7.

* subversion/libsvn_fs_fs/structure

(): Add format 7 to format list and add index files as new

elements. Describe the concept of addressing modes,

"item index" and "item type" in detail. Explain the

intention behind the new pack code.

* subversion/libsvn_fs_fs/structure-indexes

(): New file.

Detailed description of the two-way mapping between

logical and physical addresses. Also describes the

binary format used to store the index files on disk.

* subversion/libsvn_fs_fs/index.*

(): New files.

Add a FSFS-private API for both indexes providing the

mapping functionality as defined in structure-indexes.

* subversion/libsvn_fs_fs/rev_file.*

(): New files.

Provide a FSFS-private API that combines the 3 physical

files (rev/pack, 2 x index) into a common data structure

and ensures that they are being accessed consistently wrt.

pack state. This is necessary when there is an svnadmin

pack running in the background while we read the repo.

* subversion/libsvn_fs_fs/id.*

(): Replace OFFSET with ITEM_INDEX.

(svn_fs_fs__id_create_root): New utility function enabled by

all root nodes having the same

item index.

* subversion/libsvn_fs_fs/util.*

(): Add new path constructors for index files. Move item

to offset mapping and rev / pack file open functions

to their respective new API files.

* subversion/libsvn_fs_fs/low_level.*

(): Replace OFFSET with ITEM_INDEX.

* subversion/libsvn_fs_fs/fs.h

(PATH_*): Path constants for the new index files.

(CONFIG_*): New fsfs.conf configuration options.

(SVN_FS_FS__FORMAT_NUMBER): Bump format number.

(*_t): Replace OFFSET with ITEM_INDEX.

(fs_fs_data_t): Add config settings and caches for the various

index elements.

* subversion/libsvn_fs_fs/caching.c

(svn_fs_fs__initialize_caches): Initialize new caches.

* subversion/libsvn_fs_fs/fs.c

(initialize_fs_struct): Default to physical addressing.

* subversion/libsvn_fs_fs/fs_fs.c

(): Handle new addressing format info and config file options.

Use a different revision 0 template for format 7.

* subversion/libsvn_fs_fs/cached_data.c

(): Switch file access function to the new rev_file API.

Follow the OFFSET to ITEM_INDEX rename.

Implement the "block read" feature and make all data read

functionality use the block_read() function.

(dbg_log_access): Produce a more detailed access log in logical

addressing mode.

* subversion/libsvn_fs_fs/transaction.c

(): Provide generic item_index allocator and index writer

functions that work in both addressing modes. Also, add

wrappers streams that calculate the new low-level FVN-1a

checksums for everything we write to disk. Invoke all

of this when writing data to the proto-rev file. Finally,

predict the addressing mode of the future revision and

upgrade transactions from physical to log addressing mode

upon misprediction or for old txns.

* subversion/libsvn_fs_fs/pack.h

(svn_fs_fs__order_dir_entries): In mixed addressing mode repos,

the optimal read order depends

on the revision number.

* subversion/libsvn_fs_fs/pack.c

(): Implement the new reorder-on-pack feature that is a core

element of f7. Details of it are described at the top of

this file.

(svn_fs_fs__order_dir_entries): Select the optimal order depending

on the addressing mode.

* subversion/libsvn_fs_fs/tree.c

(fs_dir_optimal_order): Adapt to pack API change.

* subversion/libsvn_fs_fs/verify.c

(): In log. addressing mode, verify the forward and backward

indexes are consistent with one another. Verify that rev /

pack file size match the index values and that the low-level

checksums for every item in the rev / pack file match the

index data as well.

* subversion/libsvn_fs_fs/hotcopy.c

(): Copy the new index files as well, if the respective revs

use logical addressing. Also, clean them up when a shard

got packed since the last incremental hotcopy.

* subversion/libsvn_fs_fs/recovery.c

(): Adapt to new rev_file API and ID API changes.

* subversion/libsvn_fs_fs/rep-cache.c

(): Follow the OFFSET to ITEM_INDEX rename.

* subversion/tests/cmdline/svnadmin_tests.py

(): Add very limited f7 index read functionality - just enough

to manipulate the changed paths lists in f7 as well. Update

expected output patterns for f7. Restrict verify_non_utf8_paths

to pre-f7 repos as manipulating f7 repos is harder and not

relevant for the code / test coverage here.

* subversion/tests/cmdline/svntest/main.py

(is_fs_log_addressing): FSFS f7 also uses log addressing.

* subversion/tests/libsvn_fs_fs

(): svn:ignore test repositories

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

(): Write correct addressing mode info and check for the right

files in packed f7 repos. Add tests TXN format upgrade tests.

* tools/server-side/svn-rep-sharing-stats.c

(record): Follow the OFFSET to ITEM_INDEX rename.

  1. … 30 more files in changeset.
Introduce a null-pointer constant that we can use as a termination sentinel

in variable argument lists. This avoids the ugliness of writing

"(char *) NULL" all over the place.

* subversion/include/svn_types.h (SVN_VA_NULL): New symbol.

* subversion/include/svn_dirent_uri.h (svn_dirent_join_many),

subversion/include/svn_path.h (svn_path_join_many),

subversion/include/svn_xml.h (svn_xml_make_open_tag):

Update docstrings, mentioning SVN_VA_NULL as the argument list terminator.

[Lots of implementation files]: Replace typecast-NULL with SVN_VA_NULL in

function calls with variable argument lists.

  1. … 89 more files in changeset.
Cast NULL to (char *) in vararg calls to svn_dirent_join_many

and svn_path_join_many.

* subversion/libsvn_fs_fs/lock.c

(digest_path_from_digest, digest_path_from_path): here

* subversion/libsvn_fs_fs/transaction.c

(create_txn_dir, create_txn_dir_pre_1_5): here

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__path_rev_packed, svn_fs_fs__path_rev_shard,

svn_fs_fs__path_rev, svn_fs_fs__path_revprops_shard,

svn_fs_fs__path_revprops_pack_shard, svn_fs_fs__path_revprops,

svn_fs_fs__path_txn_dir, svn_fs_fs__path_txn_proto_rev,

svn_fs_fs__path_txn_proto_rev_lock, svn_fs_fs__path_node_origin): here

* subversion/libsvn_fs_x/lock.c

(digest_path_from_digest, digest_path_from_path): here

* subversion/libsvn_fs_x/transaction.c

(create_txn_dir): here

* subversion/libsvn_fs_x/util.c

(svn_fs_x__path_rev_packed, svn_fs_x__path_rev_shard,

svn_fs_x__path_revprops_shard,

svn_fs_x__path_revprops_pack_shard,

svn_fs_x__path_txn_dir, svn_fs_x__path_txn_proto_rev,

svn_fs_x__path_txn_proto_rev_lock, svn_fs_x__path_node_origin): here

* subversion/libsvn_subr/config_file.c

(svn_config__sys_config_path, svn_config_get_user_config_path): here

* subversion/libsvn_wc/adm_files.c

(svn_wc__adm_child): here

* subversion/libsvn_wc/upgrade.c

(build_lockfile_path, bump_to_29): here

* subversion/libsvn_wc/wc_db.c

(svn_wc__db_temp_wcroot_tempdir): here

* subversion/libsvn_wc/wc_db_pristine.c

(get_pristine_fname): here

* subversion/mod_authz_svn/mod_authz_svn.c

(get_access_conf): here

* subversion/tests/cmdline/entries-dump.c

(entries_dump): here

* subversion/tests/libsvn_client/client-test.c

(test_patch): here

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

(pack_filesystem, pack_even_filesystem, recover_fully_packed): here

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

(pack_filesystem, pack_even_filesystem, recover_fully_packed): here

* subversion/tests/libsvn_subr/dirent_uri-test.c

(test_dirent_join): here

* subversion/tests/libsvn_subr/path-test.c

(test_path_join): here

* subversion/tests/libsvn_wc/entries-compat.c

(test_entries_alloc, test_stubs): here

  1. … 18 more files in changeset.
[Reverted in r1597989]

Make native svn_fs_move support in FSFS dependent on a format bump

(tweak the conditional manually for testing). Fall back to ordinary

copy-with-history for older format.

Also, relax the condition on svn_fs_move source revision. If a rev

older than the txn's base revision is specified, there must have been

no changes at all to that node in meantime and neither the node nor

nor any of the parents been deleted (and later restored).

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__MIN_MOVE_SUPPORT_FORMAT): declare new feature conditional

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__supports_move): declare new private API

* subversion/libsvn_fs_fs/util.c

(svn_fs_fs__item_offset): implement

* subversion/include/svn_fs.h

(svn_fs_move): remove condition on from_root, it's now implementation

dependent just as other copy-related API are

* subversion/libsvn_fs_fs/tree.c

(is_changed_node): node comparison helper

(copy_helper): fall back to copy if move is not supported;

allow moves only from txn base rev or equivalent node

  1. … 4 more files in changeset.
* subversion/bindings/javahl/native/CommitEditor.h

* subversion/bindings/javahl/native/EditorProxy.h

* subversion/bindings/javahl/native/Iterator.h

* subversion/bindings/javahl/native/LockTokenTable.h

* subversion/bindings/javahl/native/RemoteSession.h

* subversion/bindings/javahl/native/RemoteSessionContext.h

* subversion/bindings/javahl/native/RevisionRangeList.h

* subversion/bindings/javahl/native/StateReporter.h

* subversion/include/private/svn_file.h

* subversion/include/private/svn_packed_data.h

* subversion/libsvn_fs_fs/cached_data.c

* subversion/libsvn_fs_fs/cached_data.h

* subversion/libsvn_fs_fs/hotcopy.c

* subversion/libsvn_fs_fs/hotcopy.h

* subversion/libsvn_fs_fs/low_level.c

* subversion/libsvn_fs_fs/low_level.h

* subversion/libsvn_fs_fs/pack.c

* subversion/libsvn_fs_fs/pack.h

* subversion/libsvn_fs_fs/recovery.c

* subversion/libsvn_fs_fs/recovery.h

* subversion/libsvn_fs_fs/revprops.c

* subversion/libsvn_fs_fs/revprops.h

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/transaction.h

* subversion/libsvn_fs_fs/util.c

* subversion/libsvn_fs_fs/util.h

* subversion/libsvn_fs_fs/verify.c

* subversion/libsvn_fs_fs/verify.h

* subversion/libsvn_fs_x/cached_data.c

* subversion/libsvn_fs_x/cached_data.h

* subversion/libsvn_fs_x/changes.c

* subversion/libsvn_fs_x/changes.h

* subversion/libsvn_fs_x/hotcopy.c

* subversion/libsvn_fs_x/hotcopy.h

* subversion/libsvn_fs_x/index.c

* subversion/libsvn_fs_x/index.h

* subversion/libsvn_fs_x/low_level.h

* subversion/libsvn_fs_x/noderevs.c

* subversion/libsvn_fs_x/noderevs.h

* subversion/libsvn_fs_x/pack.c

* subversion/libsvn_fs_x/pack.h

* subversion/libsvn_fs_x/recovery.c

* subversion/libsvn_fs_x/recovery.h

* subversion/libsvn_fs_x/reps.c

* subversion/libsvn_fs_x/reps.h

* subversion/libsvn_fs_x/revprops.h

* subversion/libsvn_fs_x/string_table.c

* subversion/libsvn_fs_x/string_table.h

* subversion/libsvn_fs_x/transaction.c

* subversion/libsvn_fs_x/transaction.h

* subversion/libsvn_fs_x/util.h

* subversion/libsvn_ra_svn/deprecated.c

* subversion/libsvn_subr/compress.c

* subversion/libsvn_subr/file.c

* subversion/libsvn_subr/packed_data.c

* subversion/libsvn_subr/prefix_string.c

* subversion/libsvn_subr/utf8proc/utf8proc.c

* subversion/libsvn_subr/utf8proc/utf8proc.h

* subversion/libsvn_subr/utf8proc/utf8proc_data.c

* subversion/libsvn_subr/utf8proc.c

* subversion/tests/libsvn_fs_x/string-table-test.c

* subversion/tests/libsvn_subr/packed-data-test.c

* subversion/tests/libsvn_subr/prefix-string-test.c

* subversion/tests/libsvn_subr/priority-queue-test.c

(svn:eol-style): Set to native.

  1. … 63 more files in changeset.
Merged branches/fsfs-improvements into /trunk (clean merge).

Removed BRANCH-README.

  1. … 47 more files in changeset.