Checkout Tools
  • last updated 7 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Reimplement fsfs private operations required by `svnfsfs` (stats, dump index,

load index) as "ioctls".

Technically we achieve this by introducing the new svn_fs_ioctl() API that

adds a generic way of performing backend-specific I/O operations.

This change serves two purposes:

- It allows us to properly expose FS-specific details and invoke FS-specific

operations everywhere without necessarily promoting them into a proper

public API (the ioctl code itself may be made either public or private,

depending on the requirements).

- It solves a potential dependency/linking problem where tools like `svnfsfs`

work through the libsvn_fs's loader, but also have to load and call private

APIs from libsvn_fs_fs thus ignoring the loader. The latter part may

potentially cause issues with the global shared state, etc. With the

patch, all such operations always go through the FS loader.

* subversion/include/svn_fs.h

(svn_fs_ioctl, SVN_FS_DECLARE_IOCTL_CODE, svn_fs_ioctl_code_t): New.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_UNRECOGNIZED_IOCTL_CODE): New error code.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now implemented as...

(SVN_FS_FS__IOCTL_GET_STATS, SVN_FS_FS__IOCTL_DUMP_INDEX,

SVN_FS_FS__IOCTL_LOAD_INDEX): ...these new ioctls, which ...

(svn_fs_fs__ioctl_get_stats_input_t, svn_fs_fs__ioctl_get_stats_output_t,

svn_fs_fs__ioctl_dump_index_input_t, svn_fs_fs__ioctl_load_index_input_t):

...use these new structures.

* subversion/libsvn_fs/fs-loader.h

(fs_library_vtable_t.ioctl, fs_vtable_t.ioctl): New vtable members.

* subversion/libsvn_fs/fs-loader.c

(svn_fs_ioctl): Implement the new API by forwarding it to an appropriate

vtable member.

* subversion/libsvn_fs_fs/fs_fs.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now declared here.

* subversion/libsvn_fs_fs/fs.c

(): Include `svn_fs_fs_private.h`.

(fs_ioctl): Implement the ioctl dispatcher for three current fsfs-specific

operations.

(fs_vtable): Initialize the `ioctl` field.

(library_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_fs/dump-index.c,

subversion/libsvn_fs_fs/load-index.c,

subversion/libsvn_fs_fs/stats.c

(): Tweak includes.

* subversion/libsvn_fs_base/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_x/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/svnfsfs/dump-index-cmd.c

(dump_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/load-index-cmd.c

(load_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/stats-cmd.c

(subcommand__stats): Invoke an appropriate svn_fs_ioctl().

* subversion/tests/libsvn_fs/fs-test.c

(test_unrecognized_ioctl): New test.

(test_funcs): Run the new test.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(get_repo_stats, dump_index, load_index): Switch to svn_fs_ioctl().

* build.conf

(svnfsfs, fs-fs-private-test): Don't link to libsvn_fs_fs.

  1. … 17 more files in changeset.
Teach 'svnfsfs stats' to show the average lengths of the representation

delta chains.

We build that info as we go: The chain length is the chain length of the

base representation we deltify against + 1. For log. addressed repositories,

we want to keep the single pass process in place. Therefore, we collect the

delta chain references and only after the whole rev / pack file was read,

will we set the counters.

Once at it, also determine and store the correct rep header size for phys.

addressed reps.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__representation_stats_t): Add the field to hold the sum of the

lengths of the delta chains.

* subversion/libsvn_fs_fs/stats.c

(rep_stats_t): Add field to hold the delta chain length.

(rep_ref_t): New temporary data structure.

(parse_representation): Set the chain length for phys. addressed reps.

(compare_representation_refs,

resolve_representation_refs): New functions to do the same for log.

addressed reps as a post-scan step.

(read_log_rev_or_packfile): Scan the file for reps as well and collect

the delta chain links. Call the above to

update the rep info afterwards.

(add_rep_stats): One more field to process in the aggregator.

* subversion/svnfsfs/stats-cmd.c

(print_rep_stats,

print_stats): Print the average delta chain lengths for these sections.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(verify_representation_stats): Have some check on the new data as well.

* subversion/tests/cmdline/svnfsfs_tests.py

(test_stats): Update expected output pattern.

  1. … 4 more files in changeset.
Fix division by zero bug when running 'svnfsfs stats' on an empty repo.

* subversion/svnfsfs/stats-cmd.c

(print_extensions_by_changes): Don't show anything if there are no entries.

(get_percentage): New utility function.

(print_extensions_by_nodes,

print_extensions_by_reps): Use the new utility to show correct percentages

even if there are only 0 length items.

* subversion/tests/cmdline/svnfsfs_tests.py

(test_stats_on_empty_repo): New test.

(test_list): Register new test.

Found by: philip

  1. … 1 more file in changeset.
Fix the lines for "length 0" in histograms produced by 'svnfsfs stats'.

They would say "-2147483648 .. < 1" instead of "0 .. < 1".

* subversion/svnfsfs/stats-cmd.c

(print_two_power): Treat anything < 1 as 0.

Tweak 'svnfsfs stats' output.

* subversion/svnfsfs/stats-cmd.c

(print_stats): Add an empty line to separate the "Reading revisions"

and "Global statistics" sections. All other sections

already have these separators.

In the 'svnfsfs stats' output, print the size ranges in histograms in a more

user friendly format like "128k .. < 256k" instead of "[2^17..2^18)".

* subversion/svnfsfs/stats-cmd.c

(print_two_power): New.

(print_histogram): Call the new function to format size ranges;

adjust the format string to accomodate larger sums.

There is no point in storing stats counters, item sizes and sums as signed

values. Use ui64 throughout.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__largest_changes_t): File sizes shall be ui64, SIZE member is

still limited to what fits into RAM.

(svn_fs_fs__large_change_info_t,

svn_fs_fs__histogram_line,

svn_fs_fs__rep_pack_stats,

svn_fs_fs__representation,

svn_fs_fs__node_stats_t,

svn_fs_fs__stats_t): Counters, sizes and sums shall be ui64.

* subversion/libsvn_fs_fs/stats.c

(rep_stats_t,

revision_info_t): Store file sizes and counters as ui64.

(get_change_count): Return count in ui64.

* subversion/svnfsfs/stats-cmd.c

(print_rep_stats,

print_largest_reps,

print_histogram,

print_extensions_by_changes,

print_extensions_by_nodes,

print_extensions_by_reps,

print_stats): Counters are unsigned now, thus use the formatter

functions for ui64.

  1. … 2 more files in changeset.
Finally, move FSFS stats logic into FSFS. This introduces the new FSFS

private API header.

* subversion/include/private/svn_fs_fs_private.h

(): New file describing the FSFS private API. The content has been

taken from the former stats-cmd.c.

* subversion/libsvn_fs_fs/stats.c

(): New file implementing svn_fs_fs__get_stats. All code has been

taken from the former stats-cmd.c.

(svn_fs_fs__get_stats): No longer static.

* subversion/svnfsfs/stats-cmd.c

(): Remove the statistics collection logic and data structures.

Remove #includes of FSFS-internal headers; use the new

svn_fs_fs_private.h instead.

  1. … 2 more files in changeset.
Minor code cleanup.

* subversion/svnfsfs/stats-cmd.c

(read_item): Remove unused parameter.

(read_log_rev_or_packfile): Update callers.

* subversion/svnfsfs/stats-cmd.c

(rep_kind_t): Use slightly better wording in commentary.

No functional change.

Finalize work on svn_fs_fs__get_stats API. Add cancellation support

to the API and make svnfsfs actually provide a standard cancellation

callback.

* subversion/svnfsfs/svnfsfs.h

(check_cancel): Declare global cancellation callback to be accessible

to all sub-commands.

* subversion/svnfsfs/svnfsfs.c

(check_cancel): New global cancellation callback. Copied over from

svnadmin.c

* subversion/svnfsfs/stats-cmd.c

(query_t): Tweak docstrings and add cancellation callback members.

(read_phys_pack_file,

read_phys_revision_file,

read_log_rev_or_packfile): Invoke the cancellation callback once

in a while.

(create_query): Store cancellation callbacks in the query struct.

(svn_fs_fs__get_stats): Add cancellation support.

(subcommand__stats): Update API caller.

Continue work on svn_fs_fs__get_stats API. We must redirect the progress

output to a progress callback.

* subversion/svnfsfs/stats-cmd.c

(query_t): Add progress callback members.

(print_progress): Remove the old implemenation.

(read_phys_pack_file,

read_phys_revision_file,

read_log_pack_file,

read_log_revision_file): Invoke progress callback instead of a local

print function.

(create_query): Store the callback in the query struct.

(svn_fs_fs__get_stats): Add progress callback parameters.

(print_progress): Reimplement with callback interface.

(subcommand__stats): Update API caller.

Follow-up to r1632681: Unbreak build.

* subversion/svnfsfs/stats-cmd.c

(svn_fs_fs__get_stats): Fix function call.

Provide an initial version of what a FSFS private API stats function

would look like. The main change over the previous read_revisions()

is that the FS struct is passed passed in instead of being created

locally and the result is only a stats struct not the full query object.

* subversion/svnfsfs/stats-cmd.c

(fs_open): Remove. Superseded by create_query().

(read_revisions): Take an existing query instance. Move the construction

code to the new construction utils.

(create_stats): New construction utility. Code mainly taken from

former read_revisions().

(create_query): New construction utility. Code taken from former

fs_open() and read_revisions().

(svn_fs_fs__get_stats): First version of the new interface function.

(subcommand__stats): Update caller.

To prevent confusing / conflicting names when moving code into FSFS,

rename the fs_t struct in stats-cmd.c to query_t and the instances

from FS to QUERY.

* subversion/svnfsfs/stats-cmd.c

(fs_t): Rename to ...

(query_t): ... this.

(open_rev_or_pack_file,

get_content,

get_cached_window,

set_cached_window,

fs_open,

find_representation,

read_rep_base,

parse_representation,

get_rep_content,

read_windows,

get_combined_window,

read_noderev,

parse_dir,

read_phys_pack_file,

read_phys_revision_file,

read_item,

read_log_rev_or_packfile,

read_log_pack_file,

read_log_revision_file,

read_revisions): Update function signatures, rename FS to QUERY and

update references to QUERY.

(subcommand__stats): Rename FS to QUERY and update references to QUERY.

Prepare 'svnfsfs stats' code to be split into a logic part within FSFS,

a UI bit in svnfsfs and an FSFS-private interface in between them.

This patch moves all stats information from the fs_t struct and function

locals into a separate svn_fs_fs__stats_t struct. All field types in that

struct are renamed to svn_fs_fs__*. No code is being moved outside the

original file, yet.

* subversion/svnfsfs/stats-cmd.c

(large_change_info_t,

largest_changes_t,

histogram_line_t,

histogram_t,

extension_info_t,

rep_pack_stats_t,

representation_stats_t,

node_stats_t): Put these into one location and rename them to ...

(svn_fs_fs__large_change_info_t,

svn_fs_fs__largest_changes_t,

svn_fs_fs__histogram_line_t,

svn_fs_fs__histogram_t,

svn_fs_fs__extension_info_t,

svn_fs_fs__rep_pack_stats_t,

svn_fs_fs__representation_stats_t,

svn_fs_fs__node_stats_t): ... these. Update members that have

renamed types.

(svn_fs_fs__stats_t): The new stats main struct. Members are taken

from fs_t or are formaer print_stats() locals.

(fs_t): Update to use a svn_fs_fs__stats_t instance now.

(initialize_largest_changes): Take the new stats type as parameter

instead of the fs_t.

(add_to_histogram): Update signature due to type rename.

(add_change): Take the new stats type as parameter instead of the fs_t.

(read_noderev): Update caller.

(read_revisions): Update initialization.

(add_rep_pack_stats,

add_rep_stats,

print_rep_stats,

print_largest_reps,

print_histogram): Update signature due to type rename.

(compare_count,

compare_node_size,

compare_rep_size,

merge_by_extension): Update local variable type names.

(get_by_extensions,

print_extensions_by_changes,

print_extensions_by_nodes,

print_extensions_by_reps,

print_histograms_by_extension): Use stats instead of fs in the signature

and update the code due to type renames.

(aggregate_stats): New function that contains the data aggregation logic

previously found in print_stats().

(print_stats): Take an stats struct instead of fs_t and just display its

contents.

(subcommand__stats): Update caller.

Switch FSFS index API functions that could actually use two different pool

parameters to the two-pool paradigm.

Although almost all current callers will pass the same (temporary) pool to

both parameters, this patch improves consistency, readability and

documentation.

* subversion/libsvn_fs_fs/index.h

(svn_fs_fs__p2l_index_lookup,

svn_fs_fs__p2l_entry_lookup,

svn_fs_fs__l2p_get_max_ids): Accept two pools instead of one.

* subversion/libsvn_fs_fs/index.c

(p2l_index_lookup): The POOL parameter is actually a SCRATCH_POOL. All

results are flat data stored in the pre-allocated

*ENTRIES result.

(svn_fs_fs__l2p_get_max_ids,

svn_fs_fs__p2l_index_lookup): Only the result array needs to be allocated

in the RESULT_POOL. It contains flat data

and no pointers.

(svn_fs_fs__p2l_entry_lookup): Use the SCRATCH_POOL with function calls

only require a temporary pool.

* subversion/libsvn_fs_fs/cached_data.c

(svn_fs_fs__check_rep,

block_read): Update caller by passing the same pool to both parameters.

That pool is already for temporaries only.

* subversion/libsvn_fs_fs/pack.c

(pack_range,

append_revision): Same.

(pack_log_addressed): Update caller, actually providing different pools.

* subversion/libsvn_fs_fs/verify.c

(compare_l2p_to_p2l_index): Same.

(compare_p2l_to_l2p_index,

compare_p2l_to_rev): Update caller by passing the same temporary pool

to both parameters.

* subversion/svnfsfs/dump-index-cmd.c

(dump_index): Same.

* subversion/svnfsfs/stats-cmd.c

(read_log_rev_or_packfile): Same.

  1. … 6 more files in changeset.
Eliminate dependency on fs_fs_data_t from svnfsfs code.

The approach is threefold. Firstly, use our own block size constants

as the API allows arbitrary values. Secondly, make the stats command

accept any repository format because we are using the generic access

functions provided FSFS. And thirdly, read the repository dimensions

once during open() and add the svn_fs_fs__* private APIs to retrieve

them where missing.

* subversion/libsvn_fs_fs/fs_fs.h

(svn_fs_fs__shard_size,

svn_fs_fs__min_unpacked_rev): Declare new private API to access data

usually read directly from fs_fs_data_t.

* subversion/libsvn_fs_fs/fs_fs.c

(svn_fs_fs__shard_size,

svn_fs_fs__min_unpacked_rev): Implement them.

* subversion/svnfsfs/svnfsfs.h

(REV_FILE_BLOCK_SIZE,

INDEX_BLOCK_SIZE): Define our rev file and index access granularities.

* subversion/svnfsfs/dump-index-cmd.c

(dump_index): Use our constants instead of an fs_fs_data_t access.

* subversion/svnfsfs/stats-cmd.c

(fs_t): Add members to store the repository dimensions.

(fs_open): Query and store the repo dimensions. Remove the obsolete

FS format check.

(read_phys_pack_file): Replace access to fs_fs_data_t with the new

fs_t members.

(read_phys_revision_file): Same. Also, fix progress for non-shared repos.

(read_item,

read_log_rev_or_packfile): Use our constants instead of an fs_fs_data_t

access.

(read_log_pack_file): Replace access to fs_fs_data_t with the new

fs_t members.

(read_log_revision_file): Same. Also, fix progress for non-shared repos.

(read_revisions): Replace access to fs_fs_data_t with the new fs_t members.

  1. … 4 more files in changeset.
Revert r1620909 as requested by zhakov.
  1. … 12 more files in changeset.
[Reverted in r1620928]

Make FSFS export the private APIs that svnfsfs comsumes.

Not much going on here, mainly moving lots of declarations

and definitions to the new svn_fs_fs_private.h header.

* build.conf

(libsvn_fs_fs): Tell msvc what to export.

* subversion/include/private/svn_fs_fs_private.h

(): New header file. Contents taken from the following headers.

* subversion/libsvn_fs_fs/fs.h

(fs_fs_shared_txn_data_t,

fs_fs_shared_data_t,

fs_fs_dag_cache_t,

fs_fs_data_t): Moved to the new header.

* subversion/libsvn_fs_fs/id.h

(svn_fs_fs__id_part_t): Same.

* subversion/libsvn_fs_fs/index.h

(SVN_FS_FS__ITEM_INDEX_*,

SVN_FS_FS__ITEM_TYPE_*,

svn_fs_fs__p2l_entry_t,

svn_fs_fs__p2l_index_lookup,

svn_fs_fs__p2l_get_max_offset,

svn_fs_fs__l2p_index_from_p2l_entries,

svn_fs_fs__p2l_index_from_p2l_entries): Same.

* subversion/libsvn_fs_fs/pack.h

(svn_fs_fs__get_packed_offset): Same.

* subversion/libsvn_fs_fs/rev_file.h

(svn_fs_fs__packed_number_stream_t,

svn_fs_fs__revision_file_t,

svn_fs_fs__open_pack_or_rev_file,

svn_fs_fs__open_pack_or_rev_file_writable,

svn_fs_fs__auto_read_footer,

svn_fs_fs__close_revision_file): Same.

* subversion/libsvn_fs_fs/transaction.h

(svn_fs_fs__add_index_data): Same.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__use_log_addressing): Same.

* subversion/libsvn_fs_fs/cached_data.c

(): Add (now) missing #include.

* subversion/svnfsfs/dump-index-cmd.c,

subversion/svnfsfs/load-index-cmd.c,

subversion/svnfsfs/stats-cmd.c:

(): #include the new header instead of the lib-internal ones.

  1. … 12 more files in changeset.
Move svnfsfs from our tools section to subversion main.

* tools/server-side/svnfsfs: Moved to ...

* subversion/svnfsfs: ... this.

* subversion/svnfsfs/dump-index-cmd.c,

subversion/svnfsfs/load-index-cmd.c,

subversion/svnfsfs/stats-cmd.c: Update relative include paths.

* build.conf

(svnfsfs): Point to the new location.

  1. … 5 more files in changeset.