Checkout Tools
  • last updated 8 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Make the dump stream parser more resilient to malformed dump streams that

do not contain \n characters at all.

Previously, we'd attempt to load the whole input into memory due to how

svn_stream_readline() is currently implemented. Doing so could potentially

choke for large files. The corresponding real-world case is where a user

(accidentally) attempts to load a huge binary file that does not contain \n

characters as the repository dump.

This is the potential cause of the OOM reported in

https://lists.apache.org/thread.html/c96eb5618ac0bf6e083345e0fdcdcf834e30913f26eabe6ada7bab62@%3Cusers.subversion.apache.org%3E

* subversion/libsvn_repos/load.c

(parse_format_version): Read the dump version string directly from

stream, with an upper limit of 80 bytes. Comment on why we don't use

svn_stream_readline() for this particular case.

(svn_repos_parse_dumpstream3): Update the call to parse_format_version().

Enable svn_repos_parse_dumpstream3() to work with incomplete parser callback

function tables. Replace the functions given by the caller as NULL with a

no-op dummy.

* subversion/include/svn_repos.h

(svn_repos_parse_dumpstream3): Add comment that caller requirements are

now relaxed.

* subversion/libsvn_repos/load.c

(dummy_handler_magic_header_record,

dummy_handler_uuid_record,

dummy_handler_new_revision_record,

dummy_handler_new_node_record,

dummy_handler_set_revision_property,

dummy_handler_set_node_property,

dummy_handler_delete_node_property,

dummy_handler_remove_node_props,

dummy_handler_set_fulltext,

dummy_handler_apply_textdelta,

dummy_handler_close_node,

dummy_handler_close_revision): Complete set of no-op dummy callbacks.

(SET_VTABLE_ENTRY): A local macro to make the next function more readable

and easier to maintain.

(complete_vtable): New function to patch up incomplate function tables.

(svn_repos_parse_dumpstream3): Patch up the function table before using it.

* subversion/tests/cmdline/svnadmin_tests.py

(load_revprops): No longer expected to fail.

  1. … 2 more files in changeset.
Remove a redundant empty write, because redundant code is annoying to read.

It sent me on a wild goose chase through the code to discover what it's for,

only to find in the end it's for nothing. The API we are providing does not

promise it, the called APIs do not require it nor guarantee to do anything

with it, and the 'svnadmin load' code does not need it. It originated as an

innocent-looking single line in r842201 and then grew to its current form in

r842708, all in the pre-1.0 era.

This code was only reached for a fulltext, never for a delta.

(svn_txdelta_parse_svndiff() treats a close after no data bytes were

received as an "early" close, and so would return an error on closing the

stream if we ever processed a zero-length delta block in this function.)

An API user would be able to see this change but should not be relying on it.

* subversion/libsvn_repos/load.c

(parse_text_block): Remove an empty write to the stream in the case of an

empty content block.

* subversion/libsvn_repos/load.c

Remove unused #include's.

(parse_text_block): Bring the doc string up to date.

Following up on r1561688, switch almost every caller of svn_stream_read() to

its successor svn_stream_read_full(). Do this to resolve many deprecation

warnings (and to avoid somebody else accidentally doing the invalid conversion

to svn_stream_read2()).

There are no functional changes as svn_stream_read() is a direct call to

svn_stream_read_full().

* subversion/libsvn_delta/svndiff.c

(read_one_byte): Update caller.

* subversion/libsvn_delta/text_delta.c

(txdelta_next_window,

tpush_write_handler,

apply_window,

svn_txdelta_send_stream): Update caller.

* subversion/libsvn_fs_base/lock.c

(svn_fs_base__get_locks): Update caller.

* subversion/libsvn_fs_fs/recovery.c

(read_handler_recover): Update caller.

* subversion/libsvn_fs_x/cached_data.c

(get_dir_contents): Update caller.

* subversion/libsvn_fs_x/pack.c

(write_reps_containers): Update caller.

* subversion/libsvn_repos/load.c

(read_key_or_val,

parse_text_block,

svn_repos_parse_dumpstream3): Update caller.

* subversion/libsvn_subr/config_file.c

(parser_getc): Update caller.

* subversion/libsvn_subr/hash.c

(svn_hash__read_entry):

* subversion/libsvn_subr/packed_data.c

(read_stream_uint,

read_stream_data,

svn_packed__data_read): Update caller.

* subversion/libsvn_subr/subst.c

(translated_stream_read,

read_handler_special): Update caller.

* subversion/libsvn_wc/adm_crawler.c

(read_handler_copy): Update caller.

* subversion/mod_dav_svn/repos.c

(deliver): Update caller.

* subversion/svnserve/serve.c

(get_file): Update caller.

* subversion/tests/libsvn_subr/spillbuf-test.c

(test_spillbuf_stream): Update caller.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_from_string,

test_stream_compressed,

test_stream_seek_stringbuf,

test_stream_seek_translated,

test_stream_compressed_empty_file): Update caller.

* subversion/tests/libsvn_wc/pristine-store-test.c

(pristine_delete_while_open): Update caller.

* subversion/tests/svn_test_fs.c

(svn_test__stream_to_string): Update caller.

* tools/server-side/fsfs-stats.c

(read_windows): Update caller.

  1. … 18 more files in changeset.
Use svn_hash_gets and svn_hash_sets.

* subversion/libsvn_repos/authz.c

* subversion/libsvn_repos/commit.c

* subversion/libsvn_repos/delta.c

* subversion/libsvn_repos/deprecated.c

* subversion/libsvn_repos/dump.c

* subversion/libsvn_repos/fs-wrap.c

* subversion/libsvn_repos/hooks.c

* subversion/libsvn_repos/load-fs-vtable.c

* subversion/libsvn_repos/load.c

* subversion/libsvn_repos/log.c

* subversion/libsvn_repos/replay.c

* subversion/libsvn_repos/reporter.c

* subversion/libsvn_repos/repos.c

* subversion/libsvn_repos/rev_hunt.c

  1. … 13 more files in changeset.
Silence size conversion warnings under Win64 where it is safe to to so

because we know the actual value ranges.

* subversion/libsvn_repos/load.c

(parse_text_block,

svn_repos_parse_dumpstream3): bufflen is limited by stream chunk size

* subversion/svn/mergeinfo-cmd.c

(mergeinfo_diagram): those text snippets are short

* subversion/tests/libsvn_subr/io-test.c

(create_test_file): size is <= 5

  1. … 2 more files in changeset.
Accept dump format v3, except for property deltas, in svndumpfilter.

Replace the hardcoded version number ("2") with a parser callback for

the dump file format version number. Used by svndumpfilter only (so

far).

Add an API option to treat text-deltas as text instead of applying

them. TRUE for svndumpfilter only.

* subversion/tests/cmdline/svndumpfilter_tests.py

(with_deltas): New test.

(test_list): Add the test.

* subversion/tests/cmdline/svndumpfilter_tests_data/simple_v3.dump

New file.

* subversion/include/svn_repos.h

(magic_header_record): New callback declaration.

(svn_repos_parse_fns3_t): Add the new callback declaration.

(svn_repos_parse_dumpstream3): Add the text-delta option.

* subversion/libsvn_repos/load.c

(parse_format_version): While we're here, put the "output" argument

first.

(svn_repos_parse_dumpstream3): Implement the text-delta-as-text option.

Pass the format version to the new callback. Don't reject v3 dumpfiles

anymore.

* subversion/svndumpfilter/main.c

(magic_header_record): New callback.

(svn_repos_parse_fns3_t): Add the new callback.

(parse_baton_initialize): Remove the hard-coded format-version output.

(do_filter): Treat text-deltas as text.

* subversion/svnrdump/load_editor.c

(magic_header_record): New callback.

(svn_rdump__load_dumpstream): Use the new callback.

* subversion/libsvn_repos/deprecated.c

(fns3_from_fns2): Set the new callback.

(svn_repos_get_fs_build_parser3): Add the text-delta option.

* subversion/libsvn_repos/load-fs-vtable.c

(magic_header_record): New callback.

(svn_repos_get_fs_build_parser4): Use the new callback.

(svn_repos_load_fs4): Add the text-delta option.

  1. … 7 more files in changeset.
Prepare to handle dump format v3 in svndumpfilter.

This revision sets up the boilerplate for backward compatibility.

Later revisions will add a new parser callback for the version number,

and a new option to allow treating text-deltas as text (instead of

applying them immediately).

* subversion/include/svn_repos.h

(svn_repos_parse_fns3_t): New struct. Tweak the callback declarations

to reflect the usual ordering in dump files.

(svn_repos_parse_dumpstream3): New API using the new struct. No

real changes yet.

(svn_repos_get_fs_build_parser4): Modify this new-in-1.8 function to

use the new struct.

(svn_repos_parse_fns2_t,

svn_repos_parse_dumpstream2,

svn_repos_get_fs_build_parser3): Deprecate.

(svn_repos_get_fs_build_parser2): Move below below the newly-

deprecated svn_repos_parse_fns2_t declaration.

* subversion/svndumpfilter/main.c

(filtering_vtable,

do_filter): Use the new API.

* subversion/svnrdump/load_editor.c

(svn_rdump__load_dumpstream): Use the new API.

* subversion/libsvn_repos/deprecated.c

(fns2_from_fns3,

fns3_from_fns2): New functions.

(svn_repos_get_fs_build_parser3): Use the new API.

* subversion/libsvn_repos/load.c

(parse_property_block,

parse_text_block,

svn_repos_parse_dumpstream3): Use the new API.

* subversion/libsvn_repos/load-fs-vtable.c

(svn_repos_get_fs_build_parser4,

svn_repos_load_fs4): Use the new API.

  1. … 5 more files in changeset.
Rename the 'svn_error_return' macro to 'svn_error_trace'.

See email thread "[RFC] Rename 'svn_error_return' to 'svn_error_trace'?" on

2011-06-23, <http://svn.haxx.se/dev/archive-2011-06/0708.shtml>.

* subversion/include/svn_error.h

(svn_error_return): Rename to svn_error_trace.

(SVN_ERR, svn_error_purge_tracing): Track the rename.

* [everywhere]: Track the rename.

  1. … 120 more files in changeset.
* subversion/libsvn_repos/load.c

(parse_format_version): Add an %s to the error message.

Finish issue #3733 ("API layering violation in repository loading

logic").

Bifurcate load.c into load.c and load-fs-vtable.c, keeping the

dumpstream parse logic in load.c, and our specific loader vtable

implementation bits in load-fs-vtable.c. This alone causes build

failures due to illegal knowledge bleed between these blocks of the

API (you know, the complaint noted in this issue), so also made the

additional changes below:

* subversion/libsvn_repos/load.c

(parse_property_block): Move mergeinfo normalization from here ...

* subversion/libsvn_repos/load-fs-vtable.c

(set_node_property): ... to here.

  1. … 1 more file in changeset.
Follow-up to r995475:

* subversion/libsvn_repos/load.c

(): Include private/svn_dep_compat.h.

(parse_property_block): Use svn_cstring_strtoui64() instead of

svn_cstring_atoi64(), properly checking for size_t overflow.

Found by: rhuijben

* subversion/libsvn_repos/load.c

(parse_format_version, svn_repos_parse_dumpstream2): Replace use of atoi()

with svn_cstring_atoi().

* subversion/libsvn_repos/load.c

(parse_property_block): Replace two more occurrences of atoi() with

svn_cstring_atoi64(), switching to 64bit because size_t isn't

guaranteed to be 32bit.

* subversion/libsvn_repos/load.c

(parse_property_block): Use svn_cstring_atoi64() instead of atoi() (switching

to 64bit because size_t isn't guaranteed to be 32bit).

Merge r985014, r985669, and r987893 from the performance branch, which make

use of the svn_ctype_is* functions more uniform.

This also adds an include to subversion/libsvn_fs_fs/fs_fs.c which was added

as part of r983430 on the performance branch. That revision is not related

to the above, and so isn't being merged here.

  1. … 17 more files in changeset.
One last issue #3020 fix: Correctly map mergeinfo revisions when loading

incremental dumps into a repository that was not empty prior to the first

load. See http://svn.haxx.se/dev/archive-2010-05/0150.shtml.

* subversion/include/private/svn_mergeinfo_private.h

(svn_mergeinfo__adjust_mergeinfo_rangelists): New.

* subversion/libsvn_subr/mergeinfo.c

(svn_mergeinfo__adjust_mergeinfo_rangelists): New.

* subversion/libsvn_repos/load.c

(renumber_mergeinfo_revs): Adjust mergeinfo older than the oldest revision

in the dump stream by the difference between the head rev of the target

repository and the current dump stream rev.

(new_revision_record): Update parse_baton.oldest_old_rev here, a bit sooner

than we did before when we set it...

(close_revision): ...here.

* subversion/tests/cmdline/svnadmin_tests.py

(dont_drop_valid_mergeinfo_during_incremental_loads): Remove comment re

XFail.

(test_list): Remove XFail from

dont_drop_valid_mergeinfo_during_incremental_loads.

  1. … 3 more files in changeset.
Upgrade calls to a deprecated API.

* subversion/bindings/javahl/native/SVNAdmin.cpp

(SVNAdmin::setRevProp):

* subversion/svnadmin/main.c

(set_revprop):

* subversion/libsvn_repos/load.c

(set_revision_property):

Use svn_fs_change_rev_prop2() instead of svn_fs_change_rev_prop().

  1. … 2 more files in changeset.
Create one notification object for the lifetime of a load, rather than each

time we need to perform a notification.

* subversion/libsvn_repos/load.c

(parse_baton): Add notification struct.

(parse_property_block, new_revision_record, maybe_add_with_history,

new_node_record, close_node, close_revision): Use the established

notification struct, rather than creating a new one.

(svn_repos_get_fs_build_parser3): Intialize the notification struct.

Rev the repos load API to take advantage of the repository notification system.

* subversion/include/svn_repos.h

(svn_node_action, svn_repos_load_uuid): Move up in the file.

(svn_repos_notify_action_t): Add several more actions.

(svn_repos_notify_t): Add several more fields needed for the load

notification.

(svn_repos_load_fs3, svn_repos_get_fs_build_parser3): New.

(svn_repos_load_fs2, svn_repos_get_fs_build_parser2): Deprecate.

* subversion/libsvn_repos/deprecated.c

(repos_notify_handler): Support the new notification actions for load

backward compat.

(svn_repos_load_fs2, svn_repos_get_fs_build_parser2): Moved here from load.c

* subversion/libsvn_repos/load.c

(parse_baton): Remove the output stream, and use the notification func and

baton.

(parse_property_block, new_revision_record, maybe_add_with_history,

new_node_record, close_node, close_revision): Emit notifications.

(svn_repos_get_fs_build_parser2, svn_repos_load_fs2): Move to deprecated.c.

(svn_repos_get_fs_build_parser3, svn_repos_load_fs3): New.

  1. … 2 more files in changeset.
* subversion/libsvn_repos/load.c

(close_revision): Fix a integer-length warning.

Fix issue #3020 mergeinfo mapping bug when loading dump with non-contiguous

revisions.

* subversion/libsvn_repos/load.c

(parse_baton): New member last_rev_mapped.

(close_revision): Set last_rev_mapped. Map any missing revisions in the

dump stream.

(svn_repos_get_fs_build_parser2): Initialize last_rev_mapped.

* subversion/tests/cmdline/svndumpfilter_tests.py

(dropped_but_not_renumbered_empty_revs): Remove comment re XFail.

(test_list): Remove XFail from dropped_but_not_renumbered_empty_revs.

  1. … 1 more file in changeset.
Fix svndumpfilter_tests.py 5 'filter mergeinfo revs outside of dump stream'.

This change reapplies part of r927243 which was reverted in r936387

and is part of the issue #3020 group of fixes.

* subversion/libsvn_repos/load.c

(parse_baton): New member oldest_old_rev.

(renumber_mergeinfo_revs): Account for the fact that svn_merge_range_t's

start_rev member is not inclusive when renumbering merge source revs.

(close_revision): Set oldest_old_rev.

(svn_repos_get_fs_build_parser2): Initialize oldest_old_rev.

* subversion/tests/cmdline/svndumpfilter_tests.py

(filter_mergeinfo_revs_outside_of_dump_stream): Remove comment about

XFail.

(test_list): Remove XFail from

filter_mergeinfo_revs_outside_of_dump_stream.

  1. … 1 more file in changeset.
A mergeinfo sanity check for svnadmin load: Stop r0 and r1 merge source revs.

* subversion/libsvn_repos/load.c

(renumber_mergeinfo_revs): More issue #3020 work; just say no to invalid

merge source revs.

Prevent svnadmin loads of partial dumps from possibly creating invalid

mergeinfo where the start range > end range.

* subversion/libsvn_repos/load.c

(renumber_mergeinfo_revs): Yet another fix for issue #3020; if we can't

remap a mergeinfos start rev, then don't attempt to remap the end

range. We might succeed and load invalid mergeinfo.

* subversion/libsvn_repos/load.c (parse_baton): Document the rev_map member.

Revert issue #3020 fix made in r927243, see

http://svn.haxx.se/dev/archive-2010-04/0475.shtml

* subversion/libsvn_repos/load.c

Barring some whitespace conflicts, this is a straightforward reversion

of r927243.

* subversion/tests/cmdline/svnadmin_tests.py

Revert r927243 with a few minor adjustments as noted below.

(reflect_dropped_renumbered_revs):

(drop_mergeinfo_outside_of_dump_stream):

Keep the typo fixes from r927243. Adjust the XFail comments to link

to the latest discussion of this issue on the dev list.

(test_list): Remove XFail from the test

dont_drop_valid_mergeinfo_during_incremental_loads. This is a test

for the regression caused by r927243, so obviously it passes now.

  1. … 1 more file in changeset.
Typo fix and indentation fix no-functional change.

* subversion/libsvn_repos/load.c

(renumber_mergeinfo_revs): Typo fix 'revsions'->'revisions'.

Maintain '2' space indentations consistently.

Fix reopened issue #3020 'Reflect dropped/renumbered revisions in

svn:mergeinfo data during svnadmin load'

Note from the future: This change was reverted in r936387.

* subversion/libsvn_repos/load.c

(parse_baton): Document REV_MAP member. Add to new members to track the

progress of loaded revisions from the dump stream.

(renumber_mergeinfo_revs): If dealing with a dumpstream that represents

only a part of a repository, then remove mergeinfo that refers to the

"other" parts which are not present. Also fix a bug where mergeinfo

pointing to the first loaded revisision was not properly mapped.

(set_node_property): If mergeinfo is found on the first loaded revision

then filter it all, it can't possibly be valid.

(close_revision): Update the new parse_baton members.

(svn_repos_get_fs_build_parser2): Initialize the new parse_baton members.

* subversion/tests/cmdline/svnadmin_tests.py

(reflect_dropped_renumbered_revs): Remove comment re XFail.

(drop_mergeinfo_outside_of_dump_stream): Fix some comment typos and

remove comment re XFail.

(test_list): Remove XFail from reflect_dropped_renumbered_revs

and drop_mergeinfo_outside_of_dump_stream.

  1. … 1 more file in changeset.