libsvn_fs_fs

Checkout Tools
  • last updated 28 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
* subversion/libsvn_fs_fs/fs.c

(fs_ioctl): Tweak the code to have a single final statement returning

an SVN_ERR_FS_UNRECOGNIZED_IOCTL_CODE error.

* subversion/libsvn_fs_fs/fs.c

(fs_ioctl): Minor refactoring. Explicitly return after handling an ioctl

in every else-if branch, instead of potentially falling through.

Follow up to r1865987: Fix another printf format bug in FSFS.

* subversion/libsvn_fs_fs/verify.c (compare_p2l_to_rev):

Cast entry::type to unsigned integer and use the %u format specifier.

Found by: danielsh

* subversion/libsvn_fs_fs/verify.c (compare_p2l_to_rev):

Fix format string that was broken on 32-bit platforms.

fsfs, fsx: Make an error message clearer to developers.

* subversion/libsvn_fs_fs/low_level.c

(svn_fs_fs__parse_footer),

* subversion/libsvn_fs_x/low_level.c

(svn_fs_x__parse_footer):

Make the output radix explicit, so the values aren't mistaken for decimal.

(Actually, I wonder why the values aren't printed in decimal to begin with.)

  1. … 1 more file in changeset.
Report a revision's size on disk.

This adds an 'svnadmin rev-size' CLI and a libsvn_fs ioctl API to report

the total size in bytes of the representation on disk of a revision,

including rev-props, excluding FSFS indexes.

$ svnadmin rev-size /path/to/repo -r1

1337 bytes in revision 1

* subversion/svnadmin/svnadmin.c

(cmd_table): Add and document the 'rev-size' command.

(revision_size,

subcommand_rev_size): New.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__ioctl_revision_size_input_t,

svn_fs_fs__ioctl_revision_size_output_t,

SVN_FS_FS__IOCTL_REVISION_SIZE): New.

* subversion/libsvn_fs_fs/fs.c

(fs_ioctl): Handle SVN_FS_FS__IOCTL_REVISION_SIZE.

* subversion/libsvn_fs_fs/fs_fs.h

* subversion/libsvn_fs_fs/stats.c

(svn_fs_fs__revision_size,

rev_size_index_entry_cb,

rev_size_baton_t): New.

* subversion/libsvn_fs_fs/revprops.h

* subversion/libsvn_fs_fs/revprops.c

(svn_fs_fs__get_revision_props_size): New.

* tools/client-side/bash_completion

(_svnadmin): Add 'rev-size'.

  1. … 3 more files in changeset.
Reimplement fsfs private operations required by `svnfsfs` (stats, dump index,

load index) as "ioctls".

Technically we achieve this by introducing the new svn_fs_ioctl() API that

adds a generic way of performing backend-specific I/O operations.

This change serves two purposes:

- It allows us to properly expose FS-specific details and invoke FS-specific

operations everywhere without necessarily promoting them into a proper

public API (the ioctl code itself may be made either public or private,

depending on the requirements).

- It solves a potential dependency/linking problem where tools like `svnfsfs`

work through the libsvn_fs's loader, but also have to load and call private

APIs from libsvn_fs_fs thus ignoring the loader. The latter part may

potentially cause issues with the global shared state, etc. With the

patch, all such operations always go through the FS loader.

* subversion/include/svn_fs.h

(svn_fs_ioctl, SVN_FS_DECLARE_IOCTL_CODE, svn_fs_ioctl_code_t): New.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_UNRECOGNIZED_IOCTL_CODE): New error code.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now implemented as...

(SVN_FS_FS__IOCTL_GET_STATS, SVN_FS_FS__IOCTL_DUMP_INDEX,

SVN_FS_FS__IOCTL_LOAD_INDEX): ...these new ioctls, which ...

(svn_fs_fs__ioctl_get_stats_input_t, svn_fs_fs__ioctl_get_stats_output_t,

svn_fs_fs__ioctl_dump_index_input_t, svn_fs_fs__ioctl_load_index_input_t):

...use these new structures.

* subversion/libsvn_fs/fs-loader.h

(fs_library_vtable_t.ioctl, fs_vtable_t.ioctl): New vtable members.

* subversion/libsvn_fs/fs-loader.c

(svn_fs_ioctl): Implement the new API by forwarding it to an appropriate

vtable member.

* subversion/libsvn_fs_fs/fs_fs.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now declared here.

* subversion/libsvn_fs_fs/fs.c

(): Include `svn_fs_fs_private.h`.

(fs_ioctl): Implement the ioctl dispatcher for three current fsfs-specific

operations.

(fs_vtable): Initialize the `ioctl` field.

(library_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_fs/dump-index.c,

subversion/libsvn_fs_fs/load-index.c,

subversion/libsvn_fs_fs/stats.c

(): Tweak includes.

* subversion/libsvn_fs_base/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_x/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/svnfsfs/dump-index-cmd.c

(dump_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/load-index-cmd.c

(load_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/stats-cmd.c

(subcommand__stats): Invoke an appropriate svn_fs_ioctl().

* subversion/tests/libsvn_fs/fs-test.c

(test_unrecognized_ioctl): New test.

(test_funcs): Run the new test.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(get_repo_stats, dump_index, load_index): Switch to svn_fs_ioctl().

* build.conf

(svnfsfs, fs-fs-private-test): Don't link to libsvn_fs_fs.

  1. … 13 more files in changeset.
Following up on r1847572, trace the errors constructed with the new helper.

* subversion/libsvn_fs_fs/tree.c

(open_path): Wrap the err_not_directory() return values with

svn_error_trace().

Suggested by: brane

fsfs: Fix SVN-4791, an issue with the DAG open_path() that was causing

unexpected SVN_ERR_FS_NOT_DIRECTORY errors when attempting to open a path

with `open_path_node_only | open_path_allow_null` flags.

The implication of this is that certain svn_fs_closest_copy() calls could be

returing unexpected errors as well. For the end user, this means that such

errors were possible, for example, during certain `svn update`s.

The root cause of this behavior is the implementation of the optimization

within the open_path() routine. The optimization attempts to do a cache

lookup of the prospective parent directory of the path to be opened.

If the cache lookup is successful, the parent node is assumed — but not

checked — to be a directory. The absense of the check was causing

unexpected errors instead of returning `NULL` in a case where:

- open_path() is called with `open_path_node_only | open_path_allow_null`

- the path to be opened points to a non-existing path, but its parent path

is a file

- the DAG node of the "parent" file is cached

See https://lists.apache.org/thread.html/693a95b0da834387e78a7f08df2392b634397d32f37428c81c02f8c5@%3Cdev.subversion.apache.org%3E

* subversion/libsvn_fs_fs/tree.c

(err_not_directory): New helper function factored out from...

(open_path): ...here. Check the kind of the DAG node for the prospective

parent returned from cache in the `open_path_node_only` optimization.

Handle the case where it is not a directory; utilize the new helper to

construct the appropriate error.

* subversion/tests/libsvn_fs/fs-test.c

(test_closest_copy_file_replaced_with_dir): New regression test.

(test_funcs): Run new test.

  1. … 1 more file in changeset.
Let 'svnadmin recover' prune the rep-cache even if it is disabled.

Part of issue #4077: "FSFS recover should prune unborn revisions from

rep-cache.db".

Also add tests for both cases (enabled, disabled).

Pruning the rep-cache even if disabled was included as r1213716 in the

original fix for issue #4077, first released in Subversion 1.7.3, but was

reverted in r1367674 (issue #4214, recovery should not create rep-cache.db)

and so omitted from Subversion 1.8 and later series of releases.

* subversion/libsvn_fs_fs/recovery.c

(recover_body): Prune the rep-cache no matter whether it's in use.

* subversion/tests/cmdline/svnadmin_tests.py

(read_rep_cache,

check_recover_prunes_rep_cache): New functions.

(recover_prunes_rep_cache_when_enabled,

recover_prunes_rep_cache_when_disabled): New tests.

(test_list): Run them.

  1. … 1 more file in changeset.
Tweak the warning for issue 4722 "checksum fail during commit when

delta is 16K". A follow-up to r1826720 and r1827191.

* subversion/libsvn_fs_fs/cached_data.c

(rep_read_contents): Fix 'gettext' compatibility by moving the format

code macro outside the gettext macro. Use correct data type format

code. (Add punctuation too, for good measure.)

Followup to r1826720, raise a warning rather than an error

as that gives users a better chance to extract data from a

corrupt repository.

* subversion/libsvn_fs_fs/cached_data.c

(rep_read_contents): Warning rather than error.

Resolve some GCC "casting away 'const'" warnings.

* subversion/libsvn_fs_fs/dag.c

(svn_fs_fs__dag_serialize): Keep 'const' in the existing cast.

* subversion/libsvn_fs_fs/index.c,

subversion/libsvn_fs_fs/load-index.c

(compare_p2l_entry_revision): Keep 'const' in the existing cast.

* subversion/libsvn_subr/x509info.c

(CONSTANT_PAIR): Keep 'const' in the existing cast.

* tools/client-side/svn-mergeinfo-normalizer/wc_mergeinfo.c

(compare_mergeinfo): Keep 'const' in the existing cast.

  1. … 2 more files in changeset.
* subversion/libsvn_fs_fs/cached_data.c (cache_windows): Add missing SVN_ERR.

* subversion/libsvn_fs_fs/temp_serializer.h

(svn_fs_fs__raw_cached_window_t, svn_fs_fs__txdelta_cached_window_t):

Give the struct types names, for «svn diff -x-p».

Fix issue 4727 which is a bug in the fix for issue 4725.

* subversion/libsvn_fs_fs/cached_data.c

(cache_windows): Move auto_read_diff_version() call before the loop.

Fix issue 4725: zlib error in fsfs cache. Make the raw window

cache store the svndiff version of the data so that the version

is available when the data is retreived from the cache.

* subversion/libsvn_fs_fs/temp_serializer.h

(svn_fs_fs__raw_cached_window_t): Add svndiff version member.

* subversion/libsvn_fs_fs/cached_data.c

(parse_raw_window): Use version retrieved from cache.

(init_rep_state): Initialize version to the unknown value.

(cache_windows): Read version, store in cache.

* subversion/tests/cmdline/svnadmin_tests.py

(load_issue4725): Remove XFail marking.

  1. … 1 more file in changeset.
Add a checksum length verification to the FSFS checksum code. This

may have prevented the issue 4722 checksum bug by making the problem

much more visible: it would have failed lots of successful commits

that only succeeded by ignoring the length error.

* subversion/libsvn_fs_fs/cached_data.c

(rep_read_contents): This is now a READ_FULL_FN for svn_stream_t

and a short read is EOF at which point the length can be checked.

When originally written the code was a READ_FN and a short read

was not necessarily EOF.

* subversion/libsvn_fs_fs/id.c

(svn_fs_fs__id_parse): Include more information in an error message.

* subversion/libsvn_fs_fs/fs_fs.c

(write_config): Correct the generated fsfs.conf file's statement of the

default value of verify-before-commit. It is keyed not on --enable-debug

but on --enable-maintainer-mode (on Windows there is no distinction between

the two).

Fix references in FSFS comments after porting code from FSX.
Don't ignore error returns.

c: [skip]

Found using tools/dev/warn-ignored-err.sh

  1. … 1 more file in changeset.
Don't ignore error returns.

c: [skip]

Found by: danielsh

using tools/dev/warn-unused-result.sh

  1. … 7 more files in changeset.
fsfs: Use the `WITHOUT ROWID` optimization for rep-cache.db in format 8.

This optimization, introduced in SQLite 3.8.2, works well for tables that

have non-integer primary keys, such as

hash TEXT NOT NULL PRIMARY KEY

in the rep-cache.db. (See the https://sqlite.org/withoutrowid.html article

for additional details.)

A quick experiment showed a reduction of the on-disk size of the database

by ~1.75x. The lookups should also be faster, both due to the reduced

database size and due to the lesser amount of internal bsearches. This

should improve the times of new commits and `svnadmin load`, especially

for large repositories that also have large rep-cache.db files.

In order to maintain compatibility, since SQLite versions prior to 3.8.2

do not support this statement, we only start using it for fsfs format 8

repositories and simultaneously bump the minimal required SQLite version

from 3.7.12 (May 2012) to 3.8.2 (December 2013). The last step ensures that

all binaries compiled to support format 8 can work with the tables with

this optimization. Also, as the various scripts have both the minimal

and recommended (3.7.15.1) SQLite versions, we bump the recommended

version to the last 3.8.x patch version, which is 3.8.11.1.

* subversion/libsvn_fs_fs/rep-cache-db.sql

(STMT_CREATE_SCHEMA): Rename this ...

(STMT_CREATE_SCHEMA_V1): ...to this.

(STMT_CREATE_SCHEMA_V2): New, enables `WITHOUT ROWID` optimization.

(STMT_GET_REP, STMT_SET_REP, STMT_GET_REPS_FOR_RANGE,

STMT_GET_MAX_REV, STMT_DEL_REPS_YOUNGER_THAN_REV,

STMT_LOCK_REP, STMT_UNLOCK_REP):

Note that these statements work for both V1 and V2 schemas.

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__MIN_REP_CACHE_SCHEMA_V2_FORMAT): New.

* subversion/libsvn_fs_fs/rep-cache.c

(REP_CACHE_SCHEMA_FORMAT): Remove.

(open_rep_cache): Select between creating a V1 or V2 schemas based

on the format of the filesystem.

* subversion/libsvn_subr/sqlite.c

(): Bump minimum required SQLite version to 3.8.2.

* subversion/tests/cmdline/svnadmin_tests.py

(check_hotcopy_fsfs_fsx): Check if the Python's built-in SQLite version

is enough to interpret the schema of rep-cache.db, and skip the check

if it's not.

* build/generator/gen_win_dependencies.py

(_find_sqlite): Bump minimum required SQLite version to 3.8.2.

* configure.ac

(SQLITE_MINIMUM_VER): Bump to 3.8.2.

(SQLITE_RECOMMENDED_VER): Bump to 3.8.11.1.

(SQLITE_RECOMMENDED_VER_REL_YEAR): New, required to construct the

download URL which includes the release year for the newer SQLite

amalgamation versions.

(SQLITE_URL): Update the download URL.

* get-deps.sh

(SQLITE_VERSION): Bump to 3.8.11.1.

(SQLITE_VERSION_REL_YEAR): New.

(get_sqlite): Update the download URL that includes the release year

for the newer SQLite amalgamation versions.

* INSTALL

(C.12.SQLite): Bump minimum required SQLite version to 3.8.2.

(E.1.Prerequisites): Bump the minimum and recommended SQLite versions.

  1. … 6 more files in changeset.
Fix an assertion in 'svnfsfs stats' with pre-v4 FSFS repositories.

* subversion/libsvn_fs_fs/fs_fs.c

(svn_fs_fs__min_unpacked_rev): MIN_UNPACKED_REV is always well-defined but

updating it is prohibited for non-packable

repos (so we would catch bugs in our logic).

fsfs: Don't store SHA1 for property representations in format 8.

This is an extended fix for issues #4623 and #4700 using the format

groundwork (ability to have optional "-" values in the representation

strings) added in r1816347.

While we don't need to serialize SHA1 for props (it is not used), older

formats can only have representation strings that either have both the

SHA1 value *and* the uniquifier, or don't have them at all. For such

formats, both values get written to the disk only if the SHA1 is present.

We cannot omit the uniquifier, as doing so breaks svn_fs_props_changed()

for properties with shared representations, as described in the referenced

issues. Therefore, we skip writing SHA1, but only for the new format 8

where this dependency is untied and we can have the uniquifier written

to the disk without the SHA1.

* subversion/libsvn_fs_fs/transaction.c

(write_final_rev): Don't store SHA1 for property representations

in format 8.

fsfs: Lay the groundwork for an extended fix for issues #4623 and #4700.

As per r1813898, we now store both the SHA1 and the uniquifier in the

on-disk property representation strings. The SHA1 value is not required,

but has to be stored due to an existing dependency in the serializer

where the resulting strings can either have both the SHA1 value *and*

the uniquifier, or don't have them at all.

Untie this dependency by introducing a new notation ("-") for such

optional values, which would be supported by the new filesystem format 8.

This would allow us to skip writing SHA1, and only store the uniquifier

in the representation strings for the new formats.

See https://lists.apache.org/thread.html/d282f27c1260c620fe5deb7c9976f4c05bfb34d5156dee1fa6dad644@%3Cdev.subversion.apache.org%3E

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__MIN_REP_STRING_OPTIONAL_VALUES_FORMAT): New.

* subversion/libsvn_fs_fs/structure

(Revision file format): Describe the format extension.

* subversion/libsvn_fs_fs/low_level.c

(format_uniquifier): New helper function, factored out from ...

(svn_fs_fs__unparse_representation): ...here. Support the "-"

notation for the absent SHA1 and uniquifier values in the new format.

Tweak the code to handle the older formats one by one. Keep the code

for the newest format in the end to simplify extending it in the future.

(svn_fs_fs__parse_representation): Handle the new "-" notation when

parsing SHA1 and uniquifier values.

* subversion/libsvn_fs_fs/low_level.c

(svn_fs_fs__parse_representation): Note a potentially unexpected side

effect of this function, as currently it modifies the input string.

* subversion/libsvn_fs_fs/low_level.c

(format_digest): Remove the `is_null` parameter, which is always passed

in as FALSE by the three calling sites in ...

(svn_fs_fs__unparse_representation): ...this function. Note that for

the SHA1 digest this parameter depends on the `has_sha1` field, but

the code above tests for the (!rep->has_sha1) case, and thus the

actual value of the parameter is always FALSE.

Fix issue #4700 for property changes.

Coincidentally, this commit also fixes issue #4623 for FSFS.

If rep-sharing is disabled, switching from svn_fs_props_different()

to svn_fs_props_changed() fixes issue #4700 for properties, similar

to how r1813794 fixed it for file content.

However, if rep-sharing is enabled, svn_fs_props_changed() does not work

as advertised because properties do not carry a SHA1 checksum with a

"uniquifier" which identifies the transaction they were created in.

The uniquifier is used by svn_fs_props_changed() to tell apart property

representations which share content but were created in different revisions.

To fix that problem, make FSFS write SHA1 checksums along with uniquifiers

for file properties, just as it is already done for file content.

A source code comment indicates that SHA1 checksums for property reps

were not written due to concerns over disk space. In hindsight, this was

a bad trade-off because it affected correctness of svn_fs_props_changed().

* subversion/libsvn_fs_fs/transaction.c

(svn_fs_fs__set_proplist, write_final_rev): Create uniquifiers on prop reps.

* subversion/libsvn_repos/reporter.c

(update_entry): Use svn_fs_props_changed() instead of using

svn_fs_props_different(). The reasoning is the same as for the

file content fix committed in r1813794.

* basic_tests.py

(null_prop_update_last_changed_revision): This test now passes for FSFS.

It still fails for BDB, though.

* subversion/tests/cmdline/svnadmin_tests.py

(verify_non_utf8_paths): Update test expectations. This test messes about

with revision file data and thus needs an update to keep passing.

(dump_no_op_prop_change): This test now passes as well.

  1. … 3 more files in changeset.