load-index-cmd.c

Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1700799 is being indexed.

[Combines r1698359 and r170078 into a single commit for better review.]

Introduce a stream wrapper object that adds buffering support to any

readable stream. Use it on the stdin streams in our CL tools.

As it turns out, parsing data from a stdin byte-by-byte incurs a

massive overhead of 100% internal and 300% system load over a buffered

stream. 'svnadmin load-revprops' sees a 5 times speedup if all data

is in OS disc caches. This is a realistic assumption in a "final sync

and switch over to new repository" scenario.

The other 2 uses of stdin either have less data to process (svnfsfs

load-index) or parse only a small fraction of the stream (svnadmin load).

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Declare the new stream constructor API.

* subversion/libsvn_subr/stream.c

(read_handler_buffering_wrapper,

data_available_handler_buffering_wrapper,

is_buffered_handler_buffering_wrapper): Internal logic of the new

stream object.

(svn_stream_wrap_buffered_read): New constructor implementation.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Wrap the stdin stream.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Same.

* subversion/tests/libsvn_subr/stream-test.c

(struct stream_baton_t,

read_handler,

data_available_handler,

create_test_read_stream): New configurable test read stream.

(expect_line_content,

test_stream_buffered_wrapper): New test for the new wrapper stream.

(test_funcs): Register the new test.

  1. … 4 more files in changeset.
Revert r1698359 and r1700789 for rehashing.
  1. … 4 more files in changeset.
[Reverted in r1700797 and re-applied as part of r1700799.]

Remove support for mark & seek from "buffered read" stream wrapper as it

was deemed to difficult to handle. See also here:

http://mail-archives.apache.org/mod_mbox/subversion-dev/201509.mbox/%3CCAP_GPNjwhjD1Ds5%2ByYKiqr2NwzbhmnGq%3DqtB8jowBUJty7_Z3Q%40mail.gmail.com%3E

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Remove all references to mark & seek

other than saying we don't support them.

* subversion/libsvn_subr/stream.c

(buffering_stream_wrapper_baton): Remove all elements used for mark & seek.

(buffering_stream_wrapper_mark): Drop.

(read_handler_buffering_wrapper): Simplify as we always exhaust the buffer

and can completely discard it now.

(decrement_mark_count,

mark_handler_buffering_wrapper,

seek_handler_buffering_wrapper,

assert_zero_mark_count): Drop.

(svn_stream_wrap_buffered_read): Update. Allocate the buffer only one and

do that here.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Update API caller.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Update API caller.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_buffered_wrapper): Update API caller.

  1. … 4 more files in changeset.
[Reverted in r1700797 and re-applied as part of r1700799.]

Introduce a stream wrapper object that adds mark/seek support to any

readable stream. Use it on the stdin streams in our CL tools.

As it turns out, parsing data from a stdin byte-by-byte incurs a

massive overhead of 100% internal and 300% system load over a buffered

stream. 'svnadmin load-revprops' sees a 5 times speedup if all data

is in OS disc caches. This is a realistic assumption in a "final sync

and switch over to new repository" scenario.

The other 2 uses of stdin either have less data to process (svnfsfs

load-index) or parse only a small fraction of the stream (svnadmin load).

To avoid any memory usage issue due to the added buffering, svnadmin

load will not use the stream wrapper - the loader might clean up some

of the pools only once per revision.

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Declare the new stream constructor API.

* subversion/libsvn_subr/stream.c

(buffering_stream_wrapper_baton,

buffering_stream_wrapper_mark): New data structures describing the

wrapper stream and marker states.

(read_handler_buffering_wrapper,

decrement_mark_count,

mark_handler_buffering_wrapper,

seek_handler_buffering_wrapper,

data_available_handler_buffering_wrapper,

is_buffered_handler_buffering_wrapper,

assert_zero_mark_count): Internal logic of the new stream object.

(svn_stream_wrap_buffered_read): New constructor implementation.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Wrap the stdin stream.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Same.

* subversion/tests/libsvn_subr/stream-test.c

(struct stream_baton_t,

read_handler,

data_available_handler,

create_test_read_stream): New configurable test read stream.

(expect_line_content,

test_stream_buffered_wrapper): New test for the new wrapper stream.

(test_funcs): Register the new test.

  1. … 4 more files in changeset.
Fix an inconsistency between 'svnfsfs dump-index' and 'svnfsfs load-index'.

While the first writes decimal item numbers, the load would interpret them

as hex. Read them as decimal now.

* subversion/svnfsfs/load-index-cmd.c

(token_to_i64): Make the RADIX selectable.

(parse_index_line): Parse each column with the appropriate radix.

Move the FSFS index load logic into FSFS.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__load_index): Declare new API, taken from load-index-cmd.c.

Extend docstring.

* subversion/libsvn_fs_fs/load-index.c

(): New file. Update #includes.

(svn_fs_fs__load_index): Moved here from load-index-cmd.c.

* subversion/svnfsfs/load-index-cmd.c

(): Update #includes.

(svn_fs_fs__load_index): Remove here.

  1. … 2 more files in changeset.
More preparation on 'svnfsfs load-index': Move the revision format check

from the UI part into the logic part.

* subversion/svnfsfs/load-index-cmd.c

(svn_fs_fs__load_index): Check that the revision does have an index.

(load_index): Remove the check here.

Prepare the 'svnfsfs load-index' code for being split up into UI,

FSFS internal logic and a private interface in between.

This patch introduces the interface structure but does not move

any code around. We only have to factor the actual FS access

into a separate function.

* subversion/svnfsfs/load-index-cmd.c

(svn_fs_fs__load_index): New function, code taken from load_index().

(load_index): Call the new function.

Revert r1620909 as requested by zhakov.
  1. … 12 more files in changeset.
[Reverted in r1620928]

Make FSFS export the private APIs that svnfsfs comsumes.

Not much going on here, mainly moving lots of declarations

and definitions to the new svn_fs_fs_private.h header.

* build.conf

(libsvn_fs_fs): Tell msvc what to export.

* subversion/include/private/svn_fs_fs_private.h

(): New header file. Contents taken from the following headers.

* subversion/libsvn_fs_fs/fs.h

(fs_fs_shared_txn_data_t,

fs_fs_shared_data_t,

fs_fs_dag_cache_t,

fs_fs_data_t): Moved to the new header.

* subversion/libsvn_fs_fs/id.h

(svn_fs_fs__id_part_t): Same.

* subversion/libsvn_fs_fs/index.h

(SVN_FS_FS__ITEM_INDEX_*,

SVN_FS_FS__ITEM_TYPE_*,

svn_fs_fs__p2l_entry_t,

svn_fs_fs__p2l_index_lookup,

svn_fs_fs__p2l_get_max_offset,

svn_fs_fs__l2p_index_from_p2l_entries,

svn_fs_fs__p2l_index_from_p2l_entries): Same.

* subversion/libsvn_fs_fs/pack.h

(svn_fs_fs__get_packed_offset): Same.

* subversion/libsvn_fs_fs/rev_file.h

(svn_fs_fs__packed_number_stream_t,

svn_fs_fs__revision_file_t,

svn_fs_fs__open_pack_or_rev_file,

svn_fs_fs__open_pack_or_rev_file_writable,

svn_fs_fs__auto_read_footer,

svn_fs_fs__close_revision_file): Same.

* subversion/libsvn_fs_fs/transaction.h

(svn_fs_fs__add_index_data): Same.

* subversion/libsvn_fs_fs/util.h

(svn_fs_fs__use_log_addressing): Same.

* subversion/libsvn_fs_fs/cached_data.c

(): Add (now) missing #include.

* subversion/svnfsfs/dump-index-cmd.c,

subversion/svnfsfs/load-index-cmd.c,

subversion/svnfsfs/stats-cmd.c:

(): #include the new header instead of the lib-internal ones.

  1. … 12 more files in changeset.
Move svnfsfs from our tools section to subversion main.

* tools/server-side/svnfsfs: Moved to ...

* subversion/svnfsfs: ... this.

* subversion/svnfsfs/dump-index-cmd.c,

subversion/svnfsfs/load-index-cmd.c,

subversion/svnfsfs/stats-cmd.c: Update relative include paths.

* build.conf

(svnfsfs): Point to the new location.

  1. … 5 more files in changeset.