svn_error_codes.h

Checkout Tools
  • last updated 7 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Introduce 'svnadmin build-repcache' command.

Implement the 'svnadmin build-repcache' CLI and add an ioctl API for building

the representation cache.

The implementation iterates over revisions in the specified range and recursively

processes the changed nodes, starting from the corresponding revision roots.

For each changed node, it ensures that its data and property representations

exist in the rep-cache. The nodes are processed in the same order as when

committing a transaction (see write_final_rev() function in libsvn_fs_fs/transaction.c),

so that the rep-cache.db files are fully consistent.

* subversion/svnadmin/svnadmin.c

(cmd_table): Add and document the 'build-repcache' command.

(subcommand_build_repcache,

build_rep_cache,

build_rep_cache_progress_func): New.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_REP_SHARING_NOT_ALLOWED,

SVN_ERR_FS_REP_SHARING_NOT_SUPPORTED): New error codes.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__ioctl_build_rep_cache_input_t,

SVN_FS_FS__IOCTL_BUILD_REP_CACHE): New.

* subversion/libsvn_fs_fs/fs.c

(fs_ioctl): Handle SVN_FS_FS__IOCTL_BUILD_REP_CACHE.

* subversion/libsvn_fs_fs/fs_fs.h

* subversion/libsvn_fs_fs/fs_fs.c

(): Include 'low_level.h'.

(svn_fs_fs__build_rep_cache,

reindex_node,

ensure_representation_sha1): New. Iterate over revisions and recursively

process the changed nodes. For each changed node, ensure that its data

and property representations exist in the rep-cache.

* subversion/tests/cmdline/svnadmin_tests.py

(build_repcache): New test.

(test_list): Add the new test.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(): Include 'libsvn_fs_fs/rep-cache.h' and 'libsvn_fs/fs-loader.h'.

(build_rep_cache): New test.

(test_funcs): Add the new test.

* tools/client-side/bash_completion

(_svnadmin): Add the 'build-repcache' command.

Patch by: Denis Kovalchuk <denis.kovalchuk@visualsvn.com>

  1. … 8 more files in changeset.
Various spelling fixes, no functional change

* contrib/client-side/svn-clean

(--exclude): s/exluded/excluded/

* subversion/include/svn_error_codes.h

(SVN_ERR_X509_CERT_VERIFY_FAILED): s/Certficate/Certificate/

* subversion/po/de.po,

subversion/po/ja.po,

subversion/po/ko.po,

subversion/po/pl.po,

subversion/po/sv.po:

(): s/Certficate/Certificate/; s/abandonded/abandoned/

* subversion/libsvn_repos/repos.c:

(): s/abandonded/abandoned/

* subversion/libsvn_repos/repos.c,

subversion/svnadmin/svnadmin.c,

subversion/tests/libsvn_client/client-test.c:

(): s/filesytem/filesystem/

  1. … 9 more files in changeset.
Reimplement fsfs private operations required by `svnfsfs` (stats, dump index,

load index) as "ioctls".

Technically we achieve this by introducing the new svn_fs_ioctl() API that

adds a generic way of performing backend-specific I/O operations.

This change serves two purposes:

- It allows us to properly expose FS-specific details and invoke FS-specific

operations everywhere without necessarily promoting them into a proper

public API (the ioctl code itself may be made either public or private,

depending on the requirements).

- It solves a potential dependency/linking problem where tools like `svnfsfs`

work through the libsvn_fs's loader, but also have to load and call private

APIs from libsvn_fs_fs thus ignoring the loader. The latter part may

potentially cause issues with the global shared state, etc. With the

patch, all such operations always go through the FS loader.

* subversion/include/svn_fs.h

(svn_fs_ioctl, SVN_FS_DECLARE_IOCTL_CODE, svn_fs_ioctl_code_t): New.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_UNRECOGNIZED_IOCTL_CODE): New error code.

* subversion/include/private/svn_fs_fs_private.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now implemented as...

(SVN_FS_FS__IOCTL_GET_STATS, SVN_FS_FS__IOCTL_DUMP_INDEX,

SVN_FS_FS__IOCTL_LOAD_INDEX): ...these new ioctls, which ...

(svn_fs_fs__ioctl_get_stats_input_t, svn_fs_fs__ioctl_get_stats_output_t,

svn_fs_fs__ioctl_dump_index_input_t, svn_fs_fs__ioctl_load_index_input_t):

...use these new structures.

* subversion/libsvn_fs/fs-loader.h

(fs_library_vtable_t.ioctl, fs_vtable_t.ioctl): New vtable members.

* subversion/libsvn_fs/fs-loader.c

(svn_fs_ioctl): Implement the new API by forwarding it to an appropriate

vtable member.

* subversion/libsvn_fs_fs/fs_fs.h

(svn_fs_fs__get_stats, svn_fs_fs__dump_index, svn_fs_fs__load_index):

These functions are now declared here.

* subversion/libsvn_fs_fs/fs.c

(): Include `svn_fs_fs_private.h`.

(fs_ioctl): Implement the ioctl dispatcher for three current fsfs-specific

operations.

(fs_vtable): Initialize the `ioctl` field.

(library_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_fs/dump-index.c,

subversion/libsvn_fs_fs/load-index.c,

subversion/libsvn_fs_fs/stats.c

(): Tweak includes.

* subversion/libsvn_fs_base/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/libsvn_fs_x/fs.c

(library_vtable, fs_vtable): Initialize the `ioctl` field to NULL.

* subversion/svnfsfs/dump-index-cmd.c

(dump_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/load-index-cmd.c

(load_index): Invoke an appropriate svn_fs_ioctl().

* subversion/svnfsfs/stats-cmd.c

(subcommand__stats): Invoke an appropriate svn_fs_ioctl().

* subversion/tests/libsvn_fs/fs-test.c

(test_unrecognized_ioctl): New test.

(test_funcs): Run the new test.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(get_repo_stats, dump_index, load_index): Switch to svn_fs_ioctl().

* build.conf

(svnfsfs, fs-fs-private-test): Don't link to libsvn_fs_fs.

  1. … 17 more files in changeset.
Add new dirent, relpath and URI canonicalization functions that

check and report canonicalization failures.

* subversion/include/svn_dirent_uri.h: Update top-level docstring.

(svn_dirent_canonicalize_safe,

svn_relpath_canonicalize_safe,

svn_uri_canonicalize_safe): New prototypes.

* subversion/include/svn_error_codes.h

(SVN_ERR_CANONICALIZATION_FAILED): New error code.

* subversion/libsvn_subr/dirent_uri.c

(svn_dirent_canonicalize_safe,

svn_relpath_canonicalize_safe,

svn_uri_canonicalize_safe): Implement..

  1. … 2 more files in changeset.
Introduce a new error code that allows distinguishing invalid property

values with non-LF line endings.

This lays the necessary groundwork for the `svnadmin load --normalize-props`

implementation. To keep our existing API promises in functions such as

svn_repos_fs_change_node_prop(), where we say that the existing error

code (SVN_ERR_BAD_PROPERTY_VALUE) will be returned in case of any

invalid properties, we would only add the new error code to the bottom

of the chain, while still keeping the original error code first in the chain.

* subversion/include/svn_error_codes.h

(SVN_ERR_BAD_PROPERTY_VALUE_EOL): New.

* subversion/libsvn_repos/fs-wrap.c

(svn_repos__validate_prop): Return the new error code when encountering

property values with non-LF line endings. Keep the original error code

(SVN_ERR_BAD_PROPERTY_VALUE) in the top of the error chain.

  1. … 1 more file in changeset.
ra_serf: Add new bucket that wraps a readable svn_stream_t.

This new bucket would be required to make ra_serf stream svndiff deltas

without creating temporary files. (We'll take an svn_txdelta_stream_t,

convert it to a generic svn_stream_t with svn_txdelta_to_svndiff_stream(),

create a bucket around that stream, and use it as the PUT request body.

This change introduces adds the necessary bucket implementation.)

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SERF_STREAM_BUCKET_READ_ERROR): New.

* subversion/libsvn_ra_serf/ra_serf.h

(svn_ra_serf__stream_bucket_errfunc_t): Declare new callback type.

(svn_ra_serf__create_stream_bucket): Declare new function.

* subversion/libsvn_ra_serf/stream_bucket.c: New file.

(stream_bucket_ctx_t): New.

(svn_ra_serf__create_stream_bucket): New, creates the bucket and

initializes a databuf that would be used in its implementation.

Remember the passed-in errfunc callback to allow reporting extended

error information to the caller in the form of an svn_error_t.

(stream_bucket_read, stream_bucket_readline, stream_bucket_peek):

New bucket vtable members that forward everything to the databuf.

(stream_reader): New. This is the core function of the new bucket that

reads data from the svn_stream_t and possibly forwards the errors

using the stored errfunc callback.

  1. … 2 more files in changeset.
fsfs: Add initial support for LZ4 compression.

This can significantly (up to 3 times) improve the speed of commits and

other operations with large binary or incompressible files, while still

maintaining a decent compression ratio.

Our current use of zlib compression — which, depending on the protocol,

can be used multiple times — heavily affects the speed of commits with

large binary or incompressible files. According to the Squash benchmark

(https://quixdb.github.io/squash-benchmark/) and to my measurements, the

zlib compression speed with the default level is about 30-40 MiB/s, and

it doesn't matter if the file is incompressible or not.

This patch provides an alternative in the form of the LZ4 compression.

While still providing a decent compression ratio, LZ4 offers much faster

compression even than zlib with level=1, and can skip incompressible data

chunks. Presumably, LZ4 is used for on-the-fly compression in different

file systems for these reasons.

With this patch, LZ4 compression will be enabled for fsfs repositories which

specify compression-level=1 in fsfs.conf. The interoperability is implemented

by bumping the format of svndiff to 2 and the repository file system format

to 8. From the client perspective, the patch starts using LZ4 compression

only for file:// protocol, and the support/negotiation of the use of svndiff2

with LZ4 compression for http:// and svn:// can be added later.

The tests for LZ4 compression can be run with one of the following commands:

win-tests.py --fsfs-compression=1

make check FSFS_COMPRESSION=1

* subversion/include/svn_delta.h

(svn_txdelta_to_svndiff3): Update docstring.

* subversion/include/svn_error_codes.h

(SVN_ERR_LZ4_COMPRESSION_FAILED,

SVN_ERR_LZ4_DECOMPRESSION_FAILED): New error codes.

* subversion/include/private/svn_subr_private.h

(svn__compress, svn__decompress): Rename to ...

(svn__compress_zlib, svn__decompress_zlib): ..this.

(svn__compress_lz4, svn__decompress_lz4): Declare new functions.

* subversion/libsvn_subr/compress.c

(): Include LZ4 library header.

(svn__compress, svn__decompress): Rename to ...

(svn__compress_zlib, svn__decompress_zlib): ..this.

(svn__compress_lz4, svn__decompress_lz4): Implement new functions.

* subversion/libsvn_subr/packed_data.c

(write_stream_data, read_stream_data): Update usages of svn__compress()

and svn__decompress().

* subversion/libsvn_delta/svndiff.c

(SVNDIFF_V2): New.

(get_svndiff_header): Update to support svndiff2 headers.

(encode_window, decode_window, write_handler): Support svndiff2 with

LZ4 compression. Tweak the relevant comments.

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__FORMAT_NUMBER): Bump to 8.

(SVN_FS_FS__MIN_SVNDIFF2_FORMAT): New define.

* subversion/libsvn_fs_fs/fs_fs.c

(write_config): Tweak the compression-level option description.

(svn_fs_fs__create, svn_fs_fs__info_format): Update to handle the

format bump.

* subversion/libsvn_fs_fs/transaction.c

(txdelta_to_svndiff): New helper to call svn_txdelta_to_svndiff3() with

appropriate svndiff version and compression level, depending on the

file system configuration.

(rep_write_get_baton, write_container_delta_rep): Use new helper.

* subversion/libsvn_fs_fs/revprops.c

(parse_packed_revprops, repack_revprops, svn_fs_fs__copy_revprops):

Update usages of svn__compress() and svn__decompress().

* subversion/libsvn_fs_fs/structure

(Filesystem formats): Update to describe usage of svndiff2.

* subversion/tests/libsvn_subr/compress-test.c: New.

* subversion/tests/libsvn_delta/random_test.c

(DEFAULT_ITERATIONS): Increase to 60.

(do_random_test, do_random_combine_test): Test different svndiff versions

and compresssion levels.

* build.conf

(libsvn_subr): Build LZ4 library sources.

(compress-test): Add new section.

* notes/svndiff: Describe svndiff2.

* NOTICE, LICENSE: Include license for LZ4.

  1. … 16 more files in changeset.
Correct a spelling mistake in an error code name.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_AMBIUGOUS_CHECKSUM_REP): Rename to ...

(SVN_ERR_FS_AMBIGUOUS_CHECKSUM_REP): ... this.

* subversion/libsvn_fs_fs/transaction.c,

subversion/libsvn_fs_x/transaction.c

(get_shared_rep): Track rename.

  1. … 2 more files in changeset.
Make FSFS consistency no longer depend on hash algorithms.

With this patch, FSFS will use MD5 and SHA1 for consistency *checks* and

efficiently finding *potential* duplicates but never allows them to control

the repository contents and structure. The only part where we did up to now

was the rep-sharing.

We now compare the rep contents before dropping an incoming rep and replacing

it with a reference to an already existing one.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_AMBIUGOUS_CHECKSUM_REP): Declare a new error code.

* subversion/libsvn_fs_fs/cached_data.h

(svn_fs_fs__get_contents_from_file): Declare a new internal API function.

* subversion/libsvn_fs_fs/cached_data.c

(rep_read_contents): We will now call this for empty reps as well, thus

we must handle that edge case, too.

(svn_fs_fs__get_contents_from_file): Implement the new internal API such

that it will work with the existing

rep read stream functions.

* subversion/libsvn_fs_fs/transaction.c

(get_writable_proto_rev): We must now be able to re-read the file to

compare rep contents.

(get_shared_rep): Compare contents of the potential match and the new REP.

(rep_write_contents_close,

write_container_rep,

write_container_delta_rep): Update callers.

* subversion/tests/libsvn_fs/fs-test.c

(test_funcs): The test passes for FSFS now.

  1. … 4 more files in changeset.
Rollback an sqlite transaction in case we fail to COMMIT it.

Otherwise, the db connection might be left in an unusable state and can

be causing different issues, especially in case the connection is a

long-living one.

See r1741071 and the commit_with_locked_rep_cache() test.

* subversion/include/svn_error_codes.h

(SVN_ERR_SQLITE_ROLLBACK_FAILED): New error code.

* subversion/libsvn_subr/sqlite.c

(rollback_transaction): Use the new error code in case we fail to

rollback the transaction.

(svn_sqlite__finish_transaction): Rollback the transaction if the

commit fails. Add a comment explaining why this is required.

* subversion/libsvn_fs_fs/rep-cache.h

(svn_fs_fs__close_rep_cache): New function.

* subversion/libsvn_fs_fs/rep-cache.c

(svn_fs_fs__close_rep_cache): Implement this new function.

* subversion/libsvn_fs_fs/transaction.c

(svn_fs_fs_commit): Unroll the SVN_SQLITE__WITH_TXN() macro.

Handle the edge case when we fail to rollback the transaction by

immediately closing the db connection.

* subversion/tests/libsvn_fs/fs-test.c

(test_funcs): The commit_with_locked_rep_cache() test now passes.

* subversion/tests/libsvn_subr/sqlite-test.c

(test_funcs): The test_sqlite_txn_commit_busy() test now passes.

  1. … 6 more files in changeset.
Declare a new error code for use by the conflict resolver in libsvn_client.

* subversion/include/svn_error_codes.h

(SVN_ERR_CLIENT_CONFLICT_OPTION_NOT_APPLICABLE): Declare.

In FSX, introduce a denser and easier to parse on-disk representation

for property lists. In a first step, use it for node properties.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_CORRUPT_PROPLIST): Define a new parser error code.

* subversion/libsvn_fs_x/low_level.h

(svn_fs_x__parse_properties,

svn_fs_x__write_properties): Declare new internal (de-)serialization APIs.

* subversion/libsvn_fs_x/low_level.c

(svn_fs_x__parse_properties,

svn_fs_x__write_properties): Implement them.

* subversion/libsvn_fs_x/cached_data.c

(svn_fs_x__get_proplist): Call the new APIs to parse the prop lists.

Use the SVN_ERR_W macro instead of local code

for decorating error objects.

* subversion/libsvn_fs_x/transaction.c

(svn_fs_x__set_proplist,

write_hash_to_stream): Write node properties in the new format.

  1. … 4 more files in changeset.
Following up on r1714330, implement a similar limit to svnserve responses.

If --max-response-size is specified and a response exceeds that limit,

it gets truncated and the connetion will be reset. This allows us to limit

the server load caused by "don't do that" type of requests like exporting

the root of the repository instead of some project branch.

By default, this limit is 0, i.e. inactive.

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SVN_RESPONSE_SIZE): New error code.

* subversion/include/svn_ra_svn.h

(svn_ra_svn_create_conn5): Add another limit parameter.

* subversion/libsvn_ra_svn/ra_svn.h

(svn_ra_svn_conn_st): Add fields for the new data counter and its limit.

* subversion/libsvn_ra_svn/deprecated.c

(svn_ra_svn_create_conn4): Update.

* subversion/libsvn_ra_svn/marshal.c

(svn_ra_svn_create_conn5): Initialize the new struct elements.

(svn_ra_svn__reset_command_io_counters): One more counter to reset.

(check_io_limits): One more limit to check.

(writebuf_output): Count outgoing data and enforce limits.

* subversion/svnserve/server.h

(serve_params_t): Add field for the new --max-response-size option.

* subversion/svnserve/serve.c

(serve_interruptable): Pass the new option to the bumped API.

* subversion/svnserve/svnserve.c

(SVNSERVE_OPT_MAX_RESPONSE): Declare the new option.

(svnserve__options): Define and document the new option.

(sub_main): Handle the new option and pass it to the bumped API.

  1. … 7 more files in changeset.
Add the equivalent of LimitXMLRequestBody to svnserve.

The idea is simple, whenever we fill our receive buffer, we update the sum

total and compare it to some limit. Reset the counter sum at each new

command / request coming in.

If a client request exceeds the --max-request-size parameter given to

svnserve (16MB by default, twice the httpd default), the processing gets

terminated and the connection will be closed. The latter is necessary

because the protocol is stateful and we just skipped / ignored a potential

state transition.

As a result, the memory usage of a threaded server is now bound to approx.

(max-request-size + 4M) x max-threads even in high-load scenarios. On the

flip side, propsets are limited to around 15M per property by default.

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SVN_REQUEST_SIZE): New error code.

* subversion/include/svn_ra_svn.h

(svn_ra_svn_create_conn5): Bumped API, adding the new limit parameter.

(svn_ra_svn_create_conn4): Deprecate.

* subversion/libsvn_ra_svn/ra_svn.h

(svn_ra_svn_conn_st): Add fields for the data counter and its limit.

(svn_ra_svn__reset_command_io_counters): Declare a function to reset the

counter - to be called before

each new command.

* subversion/libsvn_ra_svn/deprecated.c

(svn_ra_svn_create_conn4): Implement in terms of the new API.

* subversion/libsvn_ra_svn/marshal.c

(svn_ra_svn_create_conn5): Implement like the predecessor but init the

new struct elements as well.

(svn_ra_svn__reset_command_io_counters): Implement new private API.

(check_io_limits): New function performing the actual error detection.

(readbuf_input): Count incoming data and enforce limits.

(svn_ra_svn__has_command): Be sure to count I/O per command.

(svn_ra_svn__handle_command): Same. Also handle the case that we truncated

I/O and are now in a potentially inconsistent

state.

* subversion/libsvn_ra_svn/editorp.c

(svn_ra_svn_drive_editor2): Limit the request size separately for each

editor command - not the whole editor drive.

* subversion/svnserve/server.h

(serve_params_t): Add field for the new --max-request-size option.

* subversion/svnserve/serve.c

(serve_interruptable): Pass the new option to the bumped API.

* subversion/svnserve/svnserve.c

(MAX_REQUEST_SIZE): Define the default value for the new option.

(SVNSERVE_OPT_MAX_REQUEST): Declare the new option.

(svnserve__options): Define and document the new option.

(sub_main): Handle the new option and pass it to the bumped API.

  1. … 8 more files in changeset.
Add some consistency checks to FSX' packed revprop manifest parser.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_CORRUPT_REVPROP_MANIFEST): Define new error code.

* subversion/libsvn_fs_x/revprops.c

(get_revprop_packname): Verify that the entries are in proper order

and within the shard's revision range.

  1. … 1 more file in changeset.
Complete the parsing routines for parsing git-like binary blobs. This patch

completes the parsing, but leaves out the patch application code as that

needs a bit more work before committing.

* subversion/include/svn_diff.h

(svn_diff_get_binary_diff_original_stream,

svn_diff_get_binary_diff_result_stream): New functions.

* subversion/include/svn_error_codes.h

(SVN_ERR_DIFF_UNEXPECTED_DATA): New error.

* subversion/libsvn_diff/binary_diff.c

(includes): Add svn_private_config.h and diff.h.

(base85_value,

svn_diff__base85_decode_line): New functions.

* subversion/libsvn_diff/diff.h

(svn_diff__base85_decode_line): New function.

* subversion/libsvn_diff/parse-diff.c

(includes): Add svn_private_config.h and diff.h.

(base85_baton_t): New struct.

(read_handler_base85,

close_handler_base85,

get_base85_data_stream): New functions.

(length_verify_baton_t): New struct.

(read_handler_length_verify,

close_handler_length_verify,

get_verify_length_stream): New functions.

(svn_diff_get_binary_diff_original_stream,

svn_diff_get_binary_diff_result_stream): New functions.

(parse_binary_patch): Properly set apr_file in binary patch. Switch src and

dest to the same order as used by git.

* subversion/tests/cmdline/patch_tests.py

(patch_binary_file): Update expected ordering.

  1. … 5 more files in changeset.
Reimplement svn_repos_verify_fs3() to support an arbitrary callback that

receives the information about an encountered problem and lets the caller

decide on what happens next. This supersedes the keep_going argument for

this API. A callback is optional; the behavior of this API if the callback

is not provided is equivalent to how svn_repos_verify_fs2() deals with

encountered errors. This allows seamless migration to the new API, if the

callback is not necessary. The idea is partly taken from how our existing

svn_fs_lock_many() API works with a svn_fs_lock_callback_t and passes error

information to the caller.

Immediately use the new API to provide an alternative solution for the

encountered problem with 'svnadmin verify --keep-going -q' (see r1684940)

being useless in terms that it was only giving an indication of whether a

particular repository passes the verification or not, without providing a

root cause (details) of what's wrong.

Discussion can be found in http://svn.haxx.se/dev/archive-2015-05/0141.shtml

(Subject: "Possible incompatibility of svn_repos_verify_fs2() in 1.9.0-rc1")

* subversion/include/svn_error_codes.h

(SVN_ERR_REPOS_VERIFY_FAILED): Remove this error code, as we no longer

need to send a specific error from within svn_repos_verify_fs3().

(SVN_ERR_CL_REPOS_VERIFY_FAILED): New.

* subversion/include/svn_repos.h

(svn_repos_notify_action_t): Remove svn_repos_notify_failure.

(svn_repos_notify_t): Remove 'err' field, as it is no longer needed.

(svn_repos_verify_callback_t): New optional callback type to be used with

svn_repos_verify_fs3().

(svn_repos_verify_fs3): Drop 'keep_going' argument in favor of accepting a

svn_repos_verify_callback_t. Update the docstring accordingly.

(svn_repos_verify_fs2): Update the docstring for this deprecated function.

* subversion/libsvn_repos/deprecated.c

(svn_repos_verify_fs2): Update the call to svn_repos_verify_fs3() in this

compatibility wrapper. Don't pass the verify callback.

* subversion/libsvn_repos/dump.c

(notify_verification_error): Remove; this function is no longer required.

(report_error): New helper function.

(svn_repos_verify_fs3): In case we've got a svn_repos_verify_callback_t,

call it upon receiving an FS-specific structure failure or a revision

verification failure. Delegate this action to the new report_error()

helper function. Doing so makes the caller responsible for what's going

to happen with the error. The caller can choose to store the error,

ignore it or use it in any other necessary way. If a callback returns an

error, stop the verification process and immediately return that error.

If no callback is provided, mimic the behavior of svn_repos_verify_fs2()

and return the first encountered error. Drop the logic related to error

formatting, as we no longer need it at this layer. We are going to make

a simpler replacement for it is the UI code (svnadmin.c), where it is

supposed to live.

* subversion/svnadmin/svnadmin.c

(struct repos_verify_callback_baton): New. Contains the fields that are

required to track the --keep-going errors taken from ...

(struct repos_notify_handler_baton): ...this baton. After the previous

step, this baton only contains the 'feedback_stream' field, so inline it

into every calling site.

(repos_notify_handler): Baton is now simply an svn_stream_t. Remove the

boolean-based filtering logic from this handler and drop the handling of

svn_repos_notify_failure. The latter is moved, with a bit of tweaking,

into ...

(repos_verify_callback): ...this new function, that implements a callback

for svn_repos_verify_fs3(). Depending on whether we are in --keep-going

mode or not, either dump the failure details to stderr and track them to

produce a summary, or immediately return it through the callback, thus

ending the verification process. Remember all errors in the --keep-going

mode, not only those that are associated with a particular revision.

Prior to handling the error itself, tell that we failed to verify the

revision or metadata by writing corresponding messages to stderr.

(subcommand_dump, subcommand_load, subcommand_recover, subcommand_upgrade,

subcommand_hotcopy, subcommand_pack): Inline repos_notify_handler_baton

here, as it now contains a single svn_stream_t field.

(subcommand_verify): Inline repos_notify_handler_baton here, as it now

contains a single svn_stream_t field. Avoid manipulations with boolean

fields like b->silent_errors and b->silent_running, because we no longer

need them, and the fields themselves are gone. Create a feedback stream

only in non-quiet mode, as we do in other subcommand implementations.

Create a baton for repos_verify_callback() and adjust the calling site of

svn_repos_verify_fs3(), that now needs a callback. Adjust --keep-going

summary printing to the new approach with the verification callback.

Finally, provide a simple error if we encountered at least one failure

in the --keep-going mode.

* subversion/tests/cmdline/svnadmin_tests.py

(verify_keep_going, verify_keep_going_quiet, verify_invalid_path_changes):

Adjust the expectations, because now errors go straight to stderr in both

--keep-going and ordinary modes. Where possible, make the expectations a

bit stricter by extending the lines that we check with RegexListOutput().

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(load_index, load_index_keep_going): Squash two tests into one; basically,

undo the corresponding hunk from r1683311. As we no longer have separate

keep_going mode in svn_repos_verify_fs3(), and the caller decides if the

verification continues or not, we don't have to check two different

scenarios.

(test_funcs): Track the test changes.

* subversion/tests/libsvn_fs_fs/fs-fs-fuzzy-test.c

(fuzzing_1_byte_1_rev): Adjust the call to svn_repos_verify_fs3().

[in subversion/bindings]

* javahl/src/org/apache/subversion/javahl/ReposNotifyInformation.java

(ReposNotifyInformation.Action): Remove value that used to correspond

to svn_repos_notify_failure.

* javahl/src/org/apache/subversion/javahl/ISVNRepos.java

(ISVNRepos.verify): Remove 'keepGoing' argument from the newest overload

of this method and update the docstring. This patch does not expose

the ability to have control over verification failures through a custom

callback in JavaHL bindings, but we're going to address this separately.

* javahl/src/org/apache/subversion/javahl/SVNRepos.java

(SVNRepos.verify): Adjust two overloads of this method in the ISVNRepos

interface implementation.

* javahl/native/SVNRepos.h

(SVNRepos::verify): Remove 'keepGoing' argument from declaration.

* javahl/native/SVNRepos.cpp

(SVNRepos::verify): Remove 'keepGoing' argument. Do not pass a verify

callback when calling svn_repos_verify_fs3().

* javahl/native/org_apache_subversion_javahl_SVNRepos.cpp

(Java_org_apache_subversion_javahl_SVNRepos_verify): Remove 'jkeepGoing'

argument from this JNI wrapper.

  1. … 13 more files in changeset.
Fix the behaviour of 'svnadmin verify' alias svn_repos_verify_fs3.

Without --keep-going, 'svnadmin verify' must not obscure the error

returned by the FS implementation. With --keep-going, it should not

assume that a reported error implies repository corruption, only

that the verificaton process failed.

See: http://s.apache.org/j9Z

Patch by: kotkov

me

* subversion/include/svn_error_codes.h

(SVN_ERR_REPOS_VERIFY_FAILED): Replaces SVN_ERR_REPOS_CORRUPTED.

* subversion/libsvn_fs_fs/cached_data.c (svn_fs_fs__check_rep),

subversion/libsvn_fs_x/cached_data.c (svn_fs_x__check_rep):

Return SVN_ERR_FS_CORRUPT instead of the incorrect and

obsolete SVN_ERR_REPOS_CORRUPTED.

* subversion/libsvn_repos/dump.c

(verify_fs_notify_func_baton_t): Renamed from verify_fs2_notify_func_baton_t.

There's no need to 'version' file-local symbols. All uses updated.

(verify_fs_notify_func): Renamed from verify_fs2_notify_func; see above.

(svn_repos_verify_fs3): Track metadata and revision verification failures

separately and, in keep-going mode, summarize them separately, too.

Do not obscure FS backend errors in normal mode and do not ignore

cancellations in keep-going mode.

* subversion/tests/cmdline/svnadmin_tests.py

(verify_keep_going, verify_invalid_path_changes, verify_quickly):

Adjust test case expectations.

* subversion/tests/libsvn_fs_fs/fs-fs-private-test.c

(load_index_test): Renamed from load_index.

Parametrize the test to check both normal and keep-going verification

and adjust test case expectations.

(load_index, load_index_keep_going): New test cases.

(test_funcs): Add load_index_keep_going.

  1. … 5 more files in changeset.
Following up on r1666096, introduce new error code for http status 405.

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_DAV_METHOD_NOT_ALLOWED): New error code.

(SVN_ERR_RA_SERF_SSPI_INITIALISATION_FAILED,

SVN_ERR_RA_SERF_SSL_CERT_UNTRUSTED): Deprecate errors that haven't been

used since serf 1.0.

* subversion/libsvn_ra_serf/util.c

(svn_ra_serf__error_on_status,

svn_ra_serf__unexpected_status): Use new error code.

  1. … 1 more file in changeset.
Merge the svn-auth-x509 branch to trunk.

This adds an X.509 parser which we use to display certificates via the auth

command rather than storing the details provided by serf from a connection.

* LICENSE,

NOTICE: Note that the X.509 parser is based on the parser from TropicSSL.

* build.conf

(libsvn_subr): Add svn_x509.h header to msvc-export.

(x509-test, __ALL_TESTS__): Add C tests for X.509 parser.

* subversion/include/private/svn_utf_private.h

(svn_utf__encode_ucs4_string, svn_utf__utf16_to_utf8,

svn_utf__utf32_to_utf8): New functions for converting various Unicode

character encodings needed by the X.509 parser.

* subversion/include/svn_x509.h: New header.

* subversion/include/svn_error_codes.h

(SVN_ERR_X509_CATEGORY_START): New category for errors from X.509 parser.

(SVN_ERR_ASN1_OUT_OF_DATA, SVN_ERR_ASN1_UNEXPECTED_TAG,

SVN_ERR_ASN1_INVALID_LENGTH, SVN_ERR_ASN1_LENGTH_MISMATCH,

SVN_ERR_ASN1_INVALID_DATA, SVN_ERR_X509_FEATURE_UNAVAILABLE,

SVN_ERR_X509_CERT_INVALID_PEM, SVN_ERR_X509_CERT_INVALID_FORMAT,

SVN_ERR_X509_CERT_INVALID_VERSION, SVN_ERR_X509_CERT_INVALID_SERIAL,

SVN_ERR_X509_CERT_INVALID_ALG, SVN_ERR_X509_CERT_INVALID_NAME,

SVN_ERR_X509_CERT_INVALID_DATE, SVN_ERR_X509_CERT_INVALID_PUBKEY,

SVN_ERR_X509_CERT_INVALID_SIGNATURE, SVN_ERR_X509_CERT_INVALID_EXTENSIONS,

SVN_ERR_X509_CERT_UNKNOWN_VERSION, SVN_ERR_X509_CERT_UNKNOWN_PK_ALG,

SVN_ERR_X509_CERT_SIG_MISMATCH, SVN_ERR_X509_CERT_VERIFY_FAILED):

New error codes.

* subversion/include/svn_config.h

(SVN_CONFIG_AUTHN_HOSTNAME_KEY, SVN_CONFIG_AUTHN_FINGERPRINT_KEY,

SVN_CONFIG_AUTHN_VALID_FROM_KEY, SVN_CONFIG_AUTHN_VALID_UNTIL_KEY,

SVN_CONFIG_AUTHN_ISSUER_DN_KEY): Remove constants used as keys for

storing parsed certificate info in authn files.

* subversion/libsvn_subr/x509parse.c,

subversion/libsvn_subr/x509info.c,

subversion/include/x509.h: New files for implementing the X.509 parser.

* subversion/libsvn_subr/ssl_server_trust_providers.c

(ssl_server_trust_file_first_credentials,

ssl_server_trust_file_save_credentials): Don't store/retrive parsed

details of X.509 certificates.

* subversion/libsvn_subr/utf.c

(membuf_insert_ucs4, svn_utf__utf16_to_utf8, svn_utf__utf32_to_utf8):

New functions to implement Unicode conversions.

* subversion/libsvn_subr/utf8proc.c

(encode_ucs4_string): Convert to the private function ...

(svn_utf__encode_ucs4_string): New function.

(svn_utf__glob): Update caller.

* subversion/svn/auth-cmd.c

(match_credential): Remove code to match the hostname/fingerprint since

the data isn't stored.

(show_cert): New function to drive the X.509 parser and then display

the certificate to the user.

(list_credential): Use show_cert().

* subversion/tests/libsvn_subr/utf-test.c

(test_utf_conversions, test_funcs): Add tests for new unicode character

set conversions.

* subversion/tests/libsvn_subr/x509-test.c: Add tests for X.509 parser.

[in subverison/bindings/javahl]

* native/jniwrapper/jni_base.cpp,

native/jniwrapper/jni_exception.hpp:

Add IllegalArgumentException exeption.

* native/AuthnCallback.cpp,

native/AuthnCallback.hpp,

src/org/apache/subversion/javahl/callback/AuthnCallback.java:

(AuthnCallback::SSLServerCertInfo): Update the getters and constructor to

reflect the info available from the X.509 parser.

* native/org_apache_subversion_javahl_util_ConfigLib.cpp

(build_credential): Update to feed AuthnCallback::SSLServerCertInfo the info

that is available.

(Java_org_apache_subversion_javahl_util_ConfigLib_nativeSearchCredentials):

Update the searching of the certificates to parse the certificate rather

than depending on the stored data.

* src/org/apache/subversion/javahl/SVNUtil.java

(SVNUtil.searchCredentials): Update hostnamePattern documentation.

* native/Promper.cpp

(Prompter::dispatch_ssl_server_trust_prompt): Update to reflect changes

to SSLServerCertInfo.

* src/org/apache/subversion/javahl/util/ConfigLib.java: Remove some commented

out code.

* tests/org/apache/subversion/javahl/UtilTests.java

(util_cred_ssl_server, testCredentials): Update tests as needed.

  1. … 29 more files in changeset.
Unbreak the build after the r1632907 merge.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_INVALID_GENERATION): Manually copy definition from

the revprop-caching-ng branch.

Rename potentially misleading FS index related error codes.

Because the term "item index" has a specific meaning in logical addressing,

we should not reuse it for general FS index-related errors.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_ITEM_INDEX_CORRUPTION,

SVN_ERR_FS_ITEM_INDEX_REVISION,

SVN_ERR_FS_ITEM_INDEX_OVERFLOW,

SVN_ERR_FS_ITEM_INDEX_INCONSISTENT): Rename these error codes to ...

(SVN_ERR_FS_INDEX_CORRUPTION,

SVN_ERR_FS_INDEX_REVISION,

SVN_ERR_FS_INDEX_OVERFLOW,

SVN_ERR_FS_INDEX_INCONSISTENT): ... these.

* subversion/libsvn_fs_fs/index.c

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/verify.c

* subversion/libsvn_fs_x/index.c

* subversion/libsvn_fs_x/transaction.c

* subversion/libsvn_fs_x/verify.c

(): Global search of the old error codes and replace them with the new ones.

  1. … 6 more files in changeset.
Harden FSFS parsers against corrupted ID strings.

Currently, we would simply return a NULL ID in case of invalid input.

That leads to NULL struct members in other places - which is a bad idea.

Make the ID parser follow the usual svn_error_t * return pattern.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_MALFORMED_NODEREV_ID): Declare a new error code.

* subversion/libsvn_fs_fs/id.h

(svn_fs_fs__id_copy): Change signature to return svn_error_t *.

* subversion/libsvn_fs_fs/id.c

(svn_fs_fs__id_parse): Rename to ...

(id_parse): ... this and make it static.

(svn_fs_fs__id_parse): New, error generating wrapper.

* subversion/libsvn_fs_fs/cached_data.c

(read_dir_entries): Update caller.

* subversion/libsvn_fs_fs/fs_fs.c

(svn_fs_fs__get_node_origin): Same.

* subversion/libsvn_fs_fs/low_level.c

(read_change,

svn_fs_fs__read_noderev): Same.

* subversion/libsvn_fs_fs/recovery.c

(recover_find_max_ids): Same.

  1. … 6 more files in changeset.
Remove the "checked" option from svn_mutex_t. Update all callers

and remove the regression test.

The reason is that there seems to be no portable way (e.g. via a

series of conversions) to use APR atomic CAS operations with

apr_os_thread_t.

* subversion/include/private/svn_mutex.h

(svn_mutex__init): Remove CHECKED option.

* subversion/include/svn_error_codes.h

(SVN_ERR_RECURSIVE_LOCK): Remove as it has no users anymore.

(SVN_ERR_INVALID_INPUT): Update number.

* subversion/libsvn_subr/mutex.c

(svn_mutex__t): Remove all aux. members that were used for mutex

checking only. Make sure we don't end up with an

empty struct on systems that don't support threads.

(svn_mutex__init,

svn_mutex__lock,

svn_mutex__unlock): Remove all mutex the checking code.

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

(never_reached,

lock_again,

recursive_locking): Remove test case for "checked" mutexes.

(test_funcs): Remove test from list.

* subversion/libsvn_fs_base/bdb/env.c

(bdb_init_cb): Update mutex init call.

* subversion/libsvn_fs_fs/fs.c

(fs_serialized_init): Same.

* subversion/libsvn_fs/fs-loader.c

(synchronized_initialize): Same.

* subversion/libsvn_fs_x/fs.c

(x_serialized_init): Same.

* subversion/libsvn_ra_svn/cyrus_auth.c

(sasl_mutex_alloc_cb_internal,

svn_ra_svn__sasl_common_init): Same.

* subversion/libsvn_subr/cache-inprocess.c

(svn_cache__create_inprocess): Same.

* subversion/libsvn_subr/cache-membuffer.c

(svn_cache__membuffer_cache_create,

svn_cache__create_membuffer_cache): Same.

* subversion/libsvn_subr/dso.c

(svn_dso_initialize2): Same.

* subversion/libsvn_subr/named_atomic.c

(init_thread_mutex): Same.

* subversion/libsvn_subr/object_pool.c

(svn_object_pool__create): Same.

* subversion/libsvn_subr/root_pools.c

(svn_root_pools__create): Same.

* subversion/libsvn_subr/utf.c

(svn_utf_initialize2): Same.

* subversion/svnserve/logger.c

(logger__create_for_stderr,

logger__create): Same.

* subversion/tests/svn_test_main.c

(svn_test_main): Same.

  1. … 17 more files in changeset.
While our public API does not prohibit nested locking attempts to the

same mutex (e.g. via svn_fs_freeze) and returns an error on them, our

code should always be able to ensure correct lock release. Therefore,

double release etc. should be considered a malfunction rather than an

ordinary error.

* subversion/include/svn_error_codes.h

(SVN_ERR_INVALID_UNLOCK): Unreleased and no longer used. Drop.

(SVN_ERR_INVALID_INPUT): Renumber to use contiguous error codes.

* subversion/libsvn_subr/mutex.c

(svn_mutex__unlock): Consider improper unlocking a malfunction.

Suggested by: ivan

  1. … 1 more file in changeset.
Explicitly limit the dimensions of a star delta container which allows us

to cast sizes to 32 bits.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_CONTAINER_SIZE): Declare new error code.

* subversion/libsvn_fs_x/reps.h

(svn_fs_x__reps_add): Allow for error returns.

* subversion/libsvn_fs_x/reps.c

(MAX_TEXT_BODY,

MAX_INSTRUCTIONS): Formally declare dimensional limit to the star

delta container.

(svn_fs_x__reps_add_base): Update caller. Explicitly cast index to 32 bits.

(add_new_text): Explicitly cast length values to 32 bits. They have been

limited to < 2GB be the caller already.

(svn_fs_x__reps_add): Make sure the container size stays within set limits.

* subversion/libsvn_fs_x/pack.c

(write_reps_containers): Update caller.

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

(test_reps): Same.

  1. … 4 more files in changeset.
Teach checked svn_mutex__t instances to detect unlock attempts on mutexes

that are not locked.

* subversion/include/svn_error_codes.h

(SVN_ERR_INVALID_UNLOCK): Declare new error code.

* subversion/libsvn_subr/mutex.c

(svn_mutex__unlock): Verify that there was actually a lock on this mutex.

  1. … 1 more file in changeset.
Follow-up to r1597989: Remove unused and not released error codes.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_AMBIGUOUS_MOVE, SVN_ERR_FS_INCOMPLETE_MOVE): Remove.

(SVN_ERR_FS_LOCK_OPERATION_FAILED, SVN_ERR_FS_UNSUPPORTED_TYPE):

Renumber.

Introduce FSFS expert tool 'svnfsfs'. It is intended to grow various

FSFS-specific commands in the future - like recreating repcache.db etc.

For now, it provides two commands to read (dump) and write (load)

format 7 index information. With these, corrupted repositories can

be manipulated / fixed by hand or script.

The former fsfs-stats tool becomes a sub-command as well.

* build.conf

(svnfsfs): Define new target for the new tool.

(__ALL__): Add new tool as dependency.

(__MORE__): Remove fsfs-stats dependency.

(fsfs-stats): Drop superseded target.

* subversion/include/svn_error_codes.h

(SVN_ERR_FS_UNSUPPORTED_TYPE,

SVN_ERR_INVALID_INPUT): Declare new error types to be used by the

new tool.

* tools/server-side/fsfs-stats.c

(): Removed, non-boilerplate sources moved to ./svnfsfs/stats-cmd.c

* tools/server-side/svnfsfs

(): New directory for the new tool; ignore new binary.

* tools/server-side/svnfsfs/svnfsfs.h

(): New file containing the tool globals.

* tools/server-side/svnfsfs/svnfsfs.c

(): Boilerplate command line interface, command and parameter definition.

* tools/server-side/svnfsfs/dump-index-cmd.c

(): New 'dump-index' sub-command implementation.

* tools/server-side/svnfsfs/load-index-cmd.c

(): New 'load-index' sub-command implementation.

* tools/server-side/svnfsfs/stats-cmd.c

(): New 'stats' sub-command implementation. Code taken from former

stand-alone tool.

  1. … 8 more files in changeset.
APR mutexes don't support recursive locking on all platforms.

As a result, trying to take out the same lock twice in the

same thread will cause a lock up under e.g. Linux. This patch

adds an option to svn_mutex__t that detects recursive locking

attempts in most cases and returns a proper error.

The idea is simply to store the thread ID of the lock OWNER

along the actual mutex object. If that matches the current

thread's ID, there is a violation. As the current thread

cannot race with itself and because any other thread uses a

different thread ID, setting and comparing this aux. info can

be done safely.

Also, we may allow for false negatives here since we only try

to detect code sequences that are already illegal in the first

place. We also don't make that check mandatory as access to

thread IDs and their comparison may be somewhat expensive on

some systems - which would impair the futex-like behavior that

we assume in some places like the caches.

A more detailed description has been added to the source code.

A FSFS-based test is added as that has been the origin of the

feature request.

Current users will be updated as follows. FS level locks and

library / module initialization will enable recursion detection.

Potentially runtime critical, internal use disables it.

* subversion/include/svn_error_codes.h

(SVN_ERR_RECURSIVE_LOCK): Define a new error code for invalid

locking schemes.

* subversion/include/private/svn_mutex.h

(svn_mutex__t): Our mutex is now a struct as we add aux.

data to it.

(svn_mutex__init): Add the CHECKED option.

* subversion/libsvn_subr/mutex.c

(svn_mutex__t): Define the mutex structure and and document

the aux. data extensively.

(svn_mutex__init): Update constructor.

(svn_mutex__lock): Optionally, check for recursive locking attempts

and update the associtated aux. data.

(svn_mutex__unlock): Optionally, update the aux. data for recursive

lock detection.

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

(never_reached,

lock_again): Callback functions required by the new test.

(recursive_locking): New test expecting an SVN_ERR_RECURSIVE_LOCK.

(test_funcs): Register the new test.

* subversion/libsvn_fs_base/bdb/env.c

(bdb_init_cb): Initialization code should detect recursions.

* subversion/libsvn_fs/fs-loader.c

(synchronized_initialize): Same.

* subversion/libsvn_subr/dso.c

(svn_dso_initialize2): Same.

* subversion/libsvn_fs_fs/fs.c

(fs_serialized_init): FS-level locks shall detect recursion do aid

API users.

* subversion/libsvn_fs_x/fs.c

(x_serialized_init): Same.

* subversion/libsvn_ra_svn/cyrus_auth.c

(sasl_mutex_alloc_cb_internal,

svn_ra_svn__sasl_common_init): Internal, potentially tightly used

mutexes shall not suffer the overhead.

* subversion/libsvn_subr/cache-inprocess.c

(svn_cache__create_inprocess): Same.

* subversion/libsvn_subr/cache-membuffer.c

(svn_cache__membuffer_cache_create,

svn_cache__create_membuffer_cache): Same.

* subversion/libsvn_subr/file.c

(init_handle_pool): Same.

* subversion/libsvn_subr/named_atomic.c

(init_thread_mutex): Same.

* subversion/libsvn_subr/object_pool.c

(svn_object_pool__create): Same.

* subversion/libsvn_subr/root_pools.c

(svn_root_pools__create): Same.

* subversion/libsvn_subr/utf.c

(svn_utf_initialize2): Same.

* subversion/svnserve/logger.c

(logger__create_for_stderr,

logger__create): Same.

* subversion/tests/svn_test_main.c

(svn_test_main): For best test coverage, we enable recursion detection

in our test suite main app.

  1. … 18 more files in changeset.