Checkout Tools
  • last updated 49 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix various typos detected by codespell

* libsvn_client/add.c, libsvn_client/blame.c, libsvn_client/commit_util.c,

libsvn_client/conflicts.c, libsvn_client/copy.c, libsvn_client/diff.c,

libsvn_client/export.c, libsvn_client/import.c, libsvn_client/merge.c,

libsvn_client/mergeinfo.c, libsvn_client/mergeinfo.h, libsvn_client/patch.c,

libsvn_client/switch.c, libsvn_client/update.c, libsvn_delta/branch_compat.c,

libsvn_delta/compat.c, libsvn_delta/compose_delta.c, libsvn_delta/svndiff.c,

libsvn_delta/xdelta.c, libsvn_diff/deprecated.c, libsvn_diff/diff_file.c,

libsvn_fs/fs-loader.c, libsvn_fs_base/bdb/locks-table.c, libsvn_fs_base/fs.c,

libsvn_fs_base/key-gen.c, libsvn_fs_base/notes/fs-history,

libsvn_fs_base/reps-strings.c, libsvn_fs_base/tree.c,

libsvn_fs_fs/cached_data.c, libsvn_fs_fs/hotcopy.c, libsvn_fs_fs/index.c,

libsvn_fs_fs/low_level.c, libsvn_fs_fs/pack.c, libsvn_fs_fs/structure,

libsvn_fs_fs/temp_serializer.c, libsvn_fs_fs/transaction.c,

libsvn_fs_fs/tree.c, libsvn_fs_fs/util.h, libsvn_fs_x/cached_data.c,

libsvn_fs_x/caching.c, libsvn_fs_x/hotcopy.c, libsvn_fs_x/id.h,

libsvn_fs_x/index.c, libsvn_fs_x/low_level.c, libsvn_fs_x/pack.c,

libsvn_fs_x/rev_file.c, libsvn_fs_x/structure, libsvn_fs_x/temp_serializer.c,

libsvn_fs_x/transaction.c, libsvn_fs_x/tree.c, libsvn_fs_x/util.h,

libsvn_ra/ra_loader.c, libsvn_ra/ra_loader.h, libsvn_ra/util.c,

libsvn_ra/wrapper_template.h, libsvn_ra_local/ra_plugin.c,

libsvn_ra_serf/commit.c, libsvn_ra_serf/get_file.c,

libsvn_ra_serf/property.c, libsvn_ra_serf/ra_serf.h, libsvn_ra_serf/update.c,

libsvn_ra_serf/util.c, libsvn_ra_svn/client.c, libsvn_ra_svn/cyrus_auth.c,

libsvn_ra_svn/marshal.c, libsvn_repos/authz_parse.c, libsvn_repos/commit.c,

libsvn_repos/compat.c, libsvn_repos/hooks.c, libsvn_repos/load.c,

libsvn_repos/log.c, libsvn_repos/notify.c, libsvn_repos/repos.c,

libsvn_repos/repos.h, libsvn_subr/cache-memcache.c, libsvn_subr/cmdline.c,

libsvn_subr/config_file.c, libsvn_subr/date.c, libsvn_subr/gpg_agent.c,

libsvn_subr/io.c, libsvn_subr/object_pool.c, libsvn_subr/path.c,

libsvn_subr/prompt.c, libsvn_subr/string.c, libsvn_subr/sysinfo.c,

libsvn_subr/temp_serializer.c, libsvn_subr/utf.c, libsvn_subr/utf8proc.c,

libsvn_subr/version.c, libsvn_subr/win32_xlate.c, libsvn_subr/x509parse.c,

libsvn_wc/conflicts.c, libsvn_wc/conflicts.h, libsvn_wc/copy.c,

libsvn_wc/lock.c, libsvn_wc/wc_db.c, libsvn_wc/wc_db.h,

libsvn_wc/wc_db_private.h:

(): Spelling fixes

  1. … 97 more files in changeset.
* everywhere: Run tools/dev/remove-trailing-whitespace.sh to remove

trailing whitespace before we branch 1.14.

  1. … 105 more files in changeset.
Don't ignore error returns.

c: [skip]

Found by: danielsh

using tools/dev/warn-unused-result.sh

  1. … 7 more files in changeset.
In FSFS and FSX, reject commits which cause MD5 or SHA1 collisions.

Since r1785754 the rep-cache no longer depends on MD5/SHA1 hash algorithms

alone to index content. So we can store colliding content without causing

repository corruption if the rep-cache is active, which is great.

However, similar problems still exist in (at least) the RA layer and the

working copy. Until those are fixed, rejecting content which causes a hash

collision is the safest approach and avoids the undesired consequences of

storing such content.

Rejecting such content only works if rep-sharing is enabled (the default).

Content with hash collisions can still be stored if rep-sharing is disabled

but please beware of annoying consequences before doing so.

This commit is a follow-up to r1785754 and transforms a warning message

added in that revision into a fatal error.

Related dev@ discussion starts at:

https://svn.haxx.se/dev/archive-2017-05/0053.shtml

Subject: [PATCH] reject SHA1 collisions

Message-ID: <20170509132523.GN55397@ted.stsp.name>

* subversion/libsvn_fs_fs/transaction.c,

subversion/libsvn_fs_x/transaction.c

(get_shared_rep): Treat a checksum collision with different content

as a fatal error.

* subversion/tests/libsvn_fs/fs-test.c

(flag_fs_warnings): Remove, not needed anymore.

(test_rep_sharing_strict_content_check): Adjust test expectations.

  1. … 2 more files in changeset.
Correct a spelling mistake in an error code name.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_AMBIUGOUS_CHECKSUM_REP): Rename to ...

(SVN_ERR_FS_AMBIGUOUS_CHECKSUM_REP): ... this.

* subversion/libsvn_fs_fs/transaction.c,

subversion/libsvn_fs_x/transaction.c

(get_shared_rep): Track rename.

  1. … 2 more files in changeset.
Merge transaction handling improvements from FSFS to FSX.

This merges revisions r1756266 and r1756377, manually resolve

text conflicts.

  1. … 1 more file in changeset.
Merge type and function usage improvements from FSFS to FSX.

This merges revisions r1756258, r1756264, r1756265 and r1756364.

All these needed manual text conflict resolution.

  1. … 2 more files in changeset.
Port the latest fixes (r1786445 and r1786446) from FSFS to FSX.

There was only one trivial text conflict to resolve.

  1. … 2 more files in changeset.
Port the "compare reps before sharing them" patches (r1785737, r1785738,

r1785741, r1785754) from FSFS to FSX.

Various text conflicts needed to be resolved. Adaptation was needed to

address the changed representation_t struct.

* subversion/libsvn_fs_x

* subversion/libsvn_fs_x/cached_data.c

* subversion/libsvn_fs_x/cached_data.h

* subversion/libsvn_fs_x/transaction.c:

Merged changes.

* subversion/tests/libsvn_fs/fs-test.c

(test_funcs): False rep-sharing tests pass for FSX as well, now.

  1. … 4 more files in changeset.
Port the "no sharing of dir reps" patch (r1785053) from FSFS to FSX.

Lots of text conflicts needed to be resolved while the overall code

churn is quite low.

  1. … 1 more file in changeset.
Sync FSX with FSFS:

Replicate the "--no-flush-to-disk" support, effectively merging revisions

r1736357, r1737355 and r1737356 from FSFS.

While the API glue logic could be re-used, the fsync code in FSX is mainly

based on svn_fs_x__batch_fsync_t object. So, all we have to do is to add

a flag to its constructor and update a few callers.

* subversion/libsvn_fs_x

(svn:mergeinfo): Mark revisions as "merged".

* subversion/libsvn_fs_x/batch_fsync.h

(svn_fs_x__batch_fsync_create): Add new FLUSH_TO_DISK option.

* subversion/libsvn_fs_x/batch_fsync.c

(svn_fs_x__batch_fsync_t): Add new option field.

(svn_fs_x__batch_fsync_create): Set new option field.

(svn_fs_x__batch_fsync_run): Call fsync if required by the option field.

* subversion/libsvn_fs_x/fs.h

(svn_fs_x__data_t): Add FLUSH_TO_DISK option field.

* subversion/libsvn_fs_x/fs.c

(initialize_fs_struct): Initialize the new field.

* subversion/libsvn_fs_x/fs_x.c

(svn_fs_x__write_format): fsync only if requested.

(read_global_config): New function - had been removed as empty previously

but now reads the new global option.

(svn_fs_x__open,

svn_fs_x__create_file_tree): Invoke the new function.

(svn_fs_x__set_uuid): fsync only if requested.

* subversion/libsvn_fs_x/pack.c

(pack_shard): Set the new fsync batch option.

* subversion/libsvn_fs_x/revprops.c

(svn_fs_x__set_revision_proplist): Same.

* subversion/libsvn_fs_x/transaction.c

(commit_body): Same.

* subversion/libsvn_fs_x/util.c

(svn_fs_x__write_min_unpacked_rev): fsync only if requested.

* subversion/libsvn_fs_x/util.h

(svn_fs_x__move_into_place): Update docstring.

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

(test_batch_fsync): Update caller.

  1. … 11 more files in changeset.
Sync FSX with FSFS:

Merge the sqlite handling patches r1741096, r1741200, r1741206 and r1741214

from FSFS and resolve the usual FSFS/FSX naming conflicts. Give new

functions proper FSX function names.

  1. … 3 more files in changeset.
Cleanse FSX changed paths lists code from the renmants of noderev_id.

* subversion/libsvn_fs_x/fs.h

(svn_fs_x__change_t): Remove noderev ID element entirely.

* subversion/libsvn_fs_x/transaction.h

(svn_fs_x__add_change): ID parameter is no longer used.

* subversion/libsvn_fs_x/transaction.c

(svn_fs_x__add_change): Update.

* subversion/libsvn_fs_x/tree.c

(add_change): Update caller.

(x_change_node_prop,

x_make_dir,

x_delete_node,

copy_helper,

x_make_file,

apply_textdelta,

apply_text): Update respective next caller level.

  1. … 3 more files in changeset.
In FSX, remove all code that reads and writes noderev IDs in changed

path lists.

* subversion/libsvn_fs_x/changes.c

(CHANGE_TXN_NODE): No longer needed.

(binary_change_t): Remove noderev ID element.

(append_change,

svn_fs_x__changes_get_list,

svn_fs_x__write_changes_container,

svn_fs_x__read_changes_container,

svn_fs_x__changes_get_list_func): Remove all noderev ID access for

containered changed paths lists.

* subversion/libsvn_fs_x/low_level.c

(read_change,

write_change_entry): No longer read and write noderev IDs when

reading / writing non-packed changes lists.

* subversion/libsvn_fs_x/transaction.c

(fold_change): The noderev ID is now meaningless here.

(svn_fs_x__add_change): Don't store the noderev ID anymore.

(write_final_rev): No longer update the noderev in changed paths lists.

(sanitize_changed_path_info): No longer neccessary.

(commit_body): Update caller.

  1. … 2 more files in changeset.
Sync FSX with FSFS: Merge all recent I/O-API-related changes (revisions

1719269 and 1719336) from FSFS to FSX. Resolve the usual text conflicts.

  1. … 8 more files in changeset.
Since we don't pre-alloc the next txn-current during commit in FSX anymore,

we don't need to take out the txn-current lock.

* subversion/libsvn_fs_x/transaction.c

(bump_ids_baton_t): Remove.

(bump_ids): Rename to ...

(bump_current): ... this and replace the BATON with "proper" parameters.

(commit_body): Update caller to invoke the bump function directly w/o

wrapping it into a locking section.

Finally, add checksums to non-packed revprop files as well.

All revprop data is checksummed now.

* subversion/libsvn_fs_x/revprops.h

(svn_fs_x__write_non_packed_revprops): Declare new internal API to

avoid code duplication.

* subversion/libsvn_fs_x/fs_x.c

(write_revision_zero): Use the new API to create the revprops.

* subversion/libsvn_fs_x/transaction.c

(write_final_revprop): Same.

* subversion/libsvn_fs_x/revprops.c

(verify_checksum): Factored out from read_packed_data_checksummed.

(read_non_packed_revprop): Call the new function to verify the

revprop file contents.

(read_packed_data_checksummed): Call factored out function now.

(svn_fs_x__write_non_packed_revprops): Implement the new API.

(write_non_packed_revprop): Use the new API to write revprops.

(copy_revprops): Verify revprops before packing them.

  1. … 3 more files in changeset.
In FSX, use the new proplist serialization for revprops as well.

* subversion/libsvn_fs_x/fs_x.c

(write_revision_zero): Create revprop for r0 using the new API.

* subversion/libsvn_fs_x/revprops.c

(parse_revprop,

write_non_packed_revprop,

write_packed_revprop): Call the new internal API instead of

the old hash serializer.

* subversion/libsvn_fs_x/transaction.c

(get_txn_proplist): Call the new proplist parser and make the

function signature more suitable for that.

(set_txn_proplist): Call the new serializer.

(svn_fs_x__change_txn_props): Update caller.

(write_final_revprop): Call the new serializer.

(svn_fs_x__txn_proplist): Update caller.

  1. … 2 more files in changeset.
In FSX, introduce a denser and easier to parse on-disk representation

for property lists. In a first step, use it for node properties.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_CORRUPT_PROPLIST): Define a new parser error code.

* subversion/libsvn_fs_x/low_level.h

(svn_fs_x__parse_properties,

svn_fs_x__write_properties): Declare new internal (de-)serialization APIs.

* subversion/libsvn_fs_x/low_level.c

(svn_fs_x__parse_properties,

svn_fs_x__write_properties): Implement them.

* subversion/libsvn_fs_x/cached_data.c

(svn_fs_x__get_proplist): Call the new APIs to parse the prop lists.

Use the SVN_ERR_W macro instead of local code

for decorating error objects.

* subversion/libsvn_fs_x/transaction.c

(svn_fs_x__set_proplist,

write_hash_to_stream): Write node properties in the new format.

  1. … 4 more files in changeset.
Sync'ing FSX with FSFS: Port minor optimizations.

This merges r1711582, r1715793, r1716047 and r1716974 from FSFS to FSX.

The usual text conflicts due to renaming and refactoring had to be resolved.

  1. … 3 more files in changeset.
Sync'ing FSX with FSFS: Merge error and overflow checking related changes.

This merges revisions r1711672, r1715947, r1716784, r1716973, r1717332 and

r1717334 from FSFS to FSX. The usual text conflicts due to renaming and

refactoring had to be resolved.

  1. … 7 more files in changeset.
Replace the hash-based in-repository representation of directories with one

that is tighter and easier to write as well as parse. Moreover, it allows

for entry names to be any utf8 string.

The new format is:

directory := u(entry-count) entry*

entry := name-as-utf8 NUL u(node-type) noderev-id

noderev-id := i(change-set) u(item-index)

u(x) ... unsigned int x in 7b/8b encoding

i(x) ... signed int x in 7b/8b encoding

Incremental directory representations work by appending new records and

applying them to a temporary hash when reading them. No special separator

is needed - only the entry-count information will be ignored in that case.

Deleted entries are signified by an SVN_FS_X__INVALID_CHANGE_SET.

* subversion/include/private/svn_subr_private.h

(svn__encode_int,

svn__decode_int): Declare signed int wrappers for svn__encode_uint

and svn__decode_uint, respectively.

* subversion/libsvn_subr/compress.c

(svn__encode_int,

svn__decode_int): Implement them.

* subversion/libsvn_fs_x/cached_data.c

(read_dir_entries): Remove old parser.

(parse_dir_entries): The new parser with a slightly modified signature.

We now expect and consume a DATA buffer instead of

a STREAM and we create the result instead of just

populating it.

(get_dir_contents): Update caller to provide the serialized contents,

allocated in the RESULT_POOL.

* subversion/libsvn_fs_x/transaction.c

(unparse_dir_entry,

unparse_dir_entries): Write the new, simpler format.

(svn_fs_x__set_entry): Update the representation of deleted entries.

  1. … 3 more files in changeset.
* subversion/libsvn_fs_x/transaction.c

(get_and_increment_txn_key_body): Update commentary. No functional change.

No longer guarantee eternally unique transaction IDs in FSX.

After the latest changes, we only need them to be unique among the running

transactions. We still increment them as before but allow for resetting

the counter upon system failure etc. This eliminates the need to fsync

txn-current and the txn-ID pre-allocation optimization.

Caveat: We must make sure that a reused transaction ID does not result

in cache key collisions. The are tree scenarios in which an txn ID might

be reused. First, after a system crash leaving txn-current empty or

with random contents. This is not a problem as the caching server process

will have died, too.

Second, the storage system (NAS) might have failed. In that case, the

server process must be restarted as well because any transient data may

have been lost leading to inconsistent state between the storage and the

server process. This implictly resolves the txn ID caching situation.

Third, a user might run 'svnadmin recover' or directly call the respective

FS API function while the server processes remain up. For that case, we

make the instance ID - which gets bumped upon recover - part of the cache

key.

* subversion/libsvn_fs_x/transaction.c

(get_and_txn_key,

bump_txn_key): Simplify and merge into ...

(get_and_increment_txn_key_body): ... this one. Add ability to detect

and recover from txn ID collisions.

(create_txn_dir): Simplify as there is no pre-allocation anymore and the

txn dir is already being created as part of txn ID

collision handling code.

(bump_ids): Remove the txn-current handling here as it is now self-

contained in create_txn_dir - without fsync calls.

* subversion/libsvn_fs_x/recovery.c

(reset_txn_number): New utility.

(recover_body): Always reset txn-current.

* subversion/libsvn_fs_x/caching.c

(svn_fs_x__initialize_caches): Guarantee disjoint cache keys after e.g.

'svnadmin recover' reset the txn-current.

* subversion/libsvn_fs_x/fs.h

(svn_fs_x__data_t): Txn ID pre-allocation is no longer needed.

  1. … 3 more files in changeset.
In FSX, make sure that the noderev-ids stored in a revision's change paths

list aren't the in-txn ids but the actual ones from the final revision.

Reset the noderev-ids for deleted nodes.

This fixes a long-standing FS API issue where the svn_fs_id_t reported in

svn_fs_path_change_t was invalid outside a transaction. It also eliminates

the last transaction reference from committed data, i.e. transaction IDs

no longer need to be globally unique but only between the tranactions

currently in existence.

A backport to FSFS should be straightforward.

* subversion/libsvn_fs_x/transaction.c

(write_final_rev): Update the CHANGED_PATHS as we go.

(sanitize_changed_path_info): New utility function for all paths that are

no longer available in the new revision.

(commit_body): Update caller.

* subversion/libsvn_fs_x/transaction.c

(write_final_rev): Minor simplification; just comparing change sets

covers all the txn / rev / invalid cases.

Code cleanup in FSX.

* subversion/libsvn_fs_x/util.h

(svn_fs_x__move_into_place2): Rename to ...

(svn_fs_x__move_into_place): ... this.

* subversion/libsvn_fs_x/util.c

(svn_fs_x__move_into_place2): Rename to ...

(svn_fs_x__move_into_place): ... this.

* subversion/libsvn_fs_x/revprops.c

(switch_to_new_revprop): Update caller.

* subversion/libsvn_fs_x/transaction.c

(bump_txn_key,

bump_ids): Same.

  1. … 3 more files in changeset.
Make the batch fsync object in FSX easier to use:

When we create a new file through svn_fs_x__batch_fsync_open_file,

implicitly call svn_fs_x__batch_fsync_new_path if necessary.

* subversion/libsvn_fs_x/batch_fsync.c

(internal_open_file): Test whether a new file will be created and

schedule any necessary additional fsyncs.

* subversion/libsvn_fs_x/revprops.c

(svn_fs_x__pack_revprops_shard): Remove redundant call.

* subversion/libsvn_fs_x/transaction.c

(write_final_revprop,

write_next_file): Same.

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

(test_batch_fsync): Same.

  1. … 3 more files in changeset.
In FSX, provide a move-into-place function that supports fsync batching.

Note that svn_fs_x__move_into_place2 does not do The Right Thing on Win32

atm but that is no different from the previous state. This will be

addressed soon.

* subversion/libsvn_fs_x/util.h

(svn_fs_x__move_into_place2): Declare new internal API.

* subversion/libsvn_fs_x/util.c

(svn_fs_x__move_into_place2): Implement it.

* subversion/libsvn_fs_x/transaction.c

(bump_txn_key,

bump_ids): Call the new batch-enabled move-into-place function.

  1. … 2 more files in changeset.
In FSX, release potentially large internal allocations before returning

from the respective FS API function.

It may be difficult for the FS API user to decide which of the functions

may allocate a larger amount of temporary memory. Hence, they might not

clean up their scratch pool very often. But whenever FSX *knows* that

its code involves opening multiple files or manipulating larger buffers,

it can make sure to use a sub-pool around that section with minimal added

overhead.

* subversion/libsvn_fs_x/transaction.c

(with_some_lock_file): Whenever we lock the repo, we will probably write

data. That's usally expensive and we give it a

separate sub-pool.

(svn_fs_x__change_txn_props): Properties can be large as well, therefore

make sure to clean them up asap.