subversion

Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1803210 is being indexed.

Build against the system utf8proc library by default instead of the

internal copy of the utf8proc code. Add --with-utf8proc=PREFIX|internal

to allow the user to specify either where to find utf8proc or that the

internal copy be used. The Windows build will continue to use the

internal utf8proc code.

* build/ac-macros/utf8proc.m4: New.

* aclocal.m4: Include utf8proc.m4.

* configure.ac: Call SVN_UTF8PROC.

* Makefile.in

(SVN_UTF8PROC_LIBS, SVN_UTF8PROC_INCLUDES): New.

* build.conf

(libsvn_subr): Add utf8proc to link.

(utf8proc): New.

* subversion/libsvn_subr/utf8proc/utf8proc.h: Rename to ...

* subversion/libsvn_subr/utf8proc/utf8proc_internal.h: ... this.

* subversion/libsvn_subr/utf8proc/utf8proc.c: Include svn_private_config.h

and utf8proc_internal.h, protect with SVN_INTERNAL_UTF8PROC.

* subversion/libsvn_subr/utf8proc.c: Include system utf8proc.h or internal

utf8proc.c.

* subversion/svn_private_config.hw: Define SVN_INTERNAL_UTF8PROC to

enable internal copy of utf8proc.

* subversion/tests/libsvn_subr/utf-test.c

(test_utf_fuzzy_escape): Allow for small output difference between

utf8proc 1.x and 2.x.

Split libsvn_subr/compress.c into three files.

This file contained routines for zlib and LZ4 compression and for general

data encoding, such as svn__encode_uint(). It would be better to maintain

all of these three groups separately.

* subversion/libsvn_subr/compress.c

(svn__encode_uint, svn__encode_int, svn__decode_uint, svn__decode_int):

Move to encode.c.

(svn_zlib__compiled_version, svn_zlib__runtime_version,

svnCompressBound, MIN_COMPRESS_SIZE, zlib_encode, zlib_decode,

svn__compress_zlib, svn__decompress_zlib): Move to compress_zlib.c.

(svn__compress_lz4, svn__decompress_lz4): Move to compress_lz4.c.

* subversion/libsvn_subr/encode.c,

subversion/libsvn_subr/compress_zlib.c,

subversion/libsvn_subr/compress_lz4.c: New files, copied from compress.c

to preserve history.

    • ?
    /trunk/subversion/libsvn_subr/compress_zlib.c
    • ?
    /trunk/subversion/libsvn_subr/compress_lz4.c
    • ?
    /trunk/subversion/libsvn_subr/encode.c
Revert r1803158 (split libsvn_subr/compress.c into three files) in order

to re-do the split while properly preserving history.

    • ?
    /trunk/subversion/libsvn_subr/compress.c
Following up r1803143, attempt to fix SWIG python bindings by not using

the forward declaration for svn_delta_editor_t.

Apparently, the parser in swig/header_wrappers.py treats such types as

opaque, and therefore incorrectly uses the %opaque_proxy when declaring

the corresponding proxy.

* subversion/include/svn_delta.h

(svn_delta_editor_t): Remove forward declaration.

(svn_delta_editor_t.apply_textdelta_stream): Update declaration of

this vtable member.

* docs/release-notes/1.10.html

(#svnserve-use-sasl): New.

svnserve: Make use-sasl=true a fatal error in SASL-less builds.

As a side effect, this revision also stops explicitly initializing 'min_ssf'

and 'max_ssf' in non-SASL mode.

This patch was tracked as SVN-4629.

* subversion/svnserve/serve.c

(find_repos): Check 'use-sasl' in SASL-less builds, too.

* subversion/libsvn_repos/repos.c

(create_conf): Update documentation.

* notes/api-errata/1.10/: New.

* notes/api-errata/1.10/svnserve001.txt: New.

    • ?
    /trunk/notes/api-errata/1.10/svnserve001.txt
* tools/dist/templates/rc-release-ann.ezt: Change the "known issues" link

to something that is more likely to work for 1.11 prereleases too.

On the 'shelve-checkpoint3' branch: Remove a note about the 'finish'

command being unfinished, following r1803107.

* notes/resolve-moves: A small Wording tweak for clarity in the section

on finding a particular move.

Suggested-by: julianfoad

* tools/dev/unix-build/Makefile.svn: Add lz4 support.

[ Note from the future: reverted in r1803192, and re-applied in r1803194

with the new files copied from compress.c to preserve history. ]

Split libsvn_subr/compress.c into three files.

This file contained routines for zlib and LZ4 compression and for general

data encoding, such as svn__encode_uint(). It would be better to maintain

all of these three groups separately.

* subversion/libsvn_subr/compress.c

(svn__encode_uint, svn__encode_int, svn__decode_uint, svn__decode_int):

Move to encode.c.

(svn_zlib__compiled_version, svn_zlib__runtime_version,

svnCompressBound, MIN_COMPRESS_SIZE, zlib_encode, zlib_decode,

svn__compress_zlib, svn__decompress_zlib): Move to compress_zlib.c.

(svn__compress_lz4, svn__decompress_lz4): Move to compress_lz4.c.

* subversion/libsvn_subr/encode.c,

subversion/libsvn_subr/compress_zlib.c,

subversion/libsvn_subr/compress_lz4.c: New files.

    • ?
    /trunk/subversion/libsvn_subr/compress_lz4.c
    • ?
    /trunk/subversion/libsvn_subr/encode.c
    • ?
    /trunk/subversion/libsvn_subr/compress_zlib.c
* subversion/mod_dav_svn/repos.c

(negotiate_encoding_prefs): Fix typo in the comment.

* subversion/libsvn_fs_fs/transaction.c:

(txdelta_to_svndiff): Use named constant for compression level 0.

Minor code tweak: use the default compression level constant for svndiff2.

This doesn't change the actual behavior of the code, as with svndiff2 the

compression_level argument is currently ignored, but saves the reader from

possibly reading this as "with svndiff2 we don't compress at all".

* subversion/libsvn_fs_fs/transaction.c:

(txdelta_to_svndiff): Use default compression level constant for svndiff2.

* subversion/libsvn_ra_serf/commit.c:

(negotiate_put_encoding): Use default compression level constant for svndiff2.

Stream svndiff deltas without creating temporary files during 'svn import'

over HTTP.

Creating the temporary files has a certain overhead (time to write them

and disk space), but what is more important, it prevents the client and

server from simultaneously processing the data, i.e., checksumming and

compressing it.

This can have a significant effect for large binary files, and the new

approach can be up to 2x faster. The actual improvement depends on the

configuration, but is generally more visible if the server and the client

process data with roughly the same speed. Several tests in my environment

showed that the import time was reduced from 63s to 32s, and from 8s to 5s

for two (different) large binary files.

Technically, this is achieved by teaching the delta editor driver that

is used in svn_client_import5() to use the new apply_textdelta_stream()

editor callback.

* subversion/libsvn_client/import.c

(open_txdelta_stream_baton_t): New.

(txdelta_stream_baton_t, txdelta_next_window, txdelta_md5_digest):

New. Used to create a custom txdelta stream that retargets the result

checksum into a supplied buffer. Used in open_txdelta_stream().

(send_file_contents): Move the core of this function, i.e., preparation

of the delta stream into ...

(open_txdelta_stream): ...this new svn_txdelta_stream_open_func_t

implementation, and pass the necessary arguments via baton. Use

result and scratch pools. Note that this function must be restartable,

as it can be called more than once, for example, if the HTTP server

decides to do a renegotiation, and we would be forced to resend

the whole request body.

(import_file): Use the new apply_textdelta_stream() editor callback.

Lay the groundwork that would allow ra_serf to stream svndiff deltas

without creating temporary files when working against new servers.

This patch adds a new delta editor callback, apply_textdelta_stream()

that we'll use to stream the deltas, fully implements in the ra_serf's

commit editor, but doesn't yet change the behavior of existing editor

drivers, such as svn_client_import5() or svn_client_commit6().

This requires a minor tweak to the Subversion's HTTP protocol, and

it's the reason why streaming would only work against new servers.

Currently, all PUT requests include a special header that contains the

result checksum, which is used by the server to validate the integrity

of the result after it applies the delta received over the wire. While

this approach works fine if the client first creates a temporary file with

the delta and only then starts sending it to the server (the result checksum

is calculated while preparing the temporary file), it can't be used in the

stream approach, as with it we'd need to know the result checksum _before_

we start sending data.

So we turn the existing scheme inside out, and teach mod_dav_svn to send the

result checksum in the responses to PUT requests. Then, the client would

check if the received checksum matches what it calculated, and, possibly,

return a checksum mismatch error (thus aborting the edit and the transaction).

* subversion/include/svn_delta.h

(svn_txdelta_stream_open_func_t): New callback type.

(svn_delta_editor_t): Add a forward declaration for this type.

(svn_delta_editor_t.apply_textdelta_stream): New vtable member.

* subversion/libsvn_delta/default_editor.c

(apply_textdelta_stream): Provide default implementation for this callback

that performs a fallback to apply_textdelta().

(default_editor): Extend the default instance of an svn_delta_editor_t.

* subversion/libsvn_delta/cancel.c

(apply_textdelta_stream): Implement this forwarding callback, and ...

(svn_delta_get_cancellation_editor): ...install it here.

* subversion/include/svn_dav.h

(SVN_DAV_NS_DAV_SVN_PUT_RESULT_CHECKSUM): New.

* subversion/mod_dav_svn/repos.c

(close_stream): Send the "X-SVN-Result-Fulltext-MD5" header when responding

to successful PUT requests.

* subversion/mod_dav_svn/version.c

(get_vsn_options): Advertise new capability.

* subversion/libsvn_ra_serf/ra_serf.h

(svn_ra_serf__session_t.supports_put_result_checksum): New field.

* subversion/libsvn_ra_serf/options.c

(capabilities_headers_iterator_callback): Remember if the server sends

the result checksum in the response to a successful PUT request.

* subversion/libsvn_ra_serf/commit.c

(file_context_t.svndiff_sent): New field.

(file_context_t.remote_result_checksum): New field.

(apply_textdelta): Update the comment stating that it would be nice to

get rid of temporary files for svndiff delta. Factor out the svndiff

format selection logic ...

(negotiate_put_encoding): ...into this new helper function.

(open_txdelta_baton_t): New.

(txdelta_stream_errfunc): New error function for the stream bucket.

(create_body_from_txdelta_stream): New svn_ra_serf__request_body_delegate_t

that creates the request body by opening an svn_txdelta_stream_t, turning

it into svn_stream_t and giving away a bucket wrapping around that stream.

Use it in ...

(apply_textdelta_stream): ...this new function that performs a PUT, and

streams the request body.

(put_response_ctx_t): New.

(put_response_handler): New, remembers the result checksum we received

from the server.

(close_file): Don't do a PUT if we did it in apply_textdelta_stream().

Check for a checksum mismatch using the checksum returned from the server.

(svn_ra_serf__get_commit_editor): Install the new apply_textdelta_stream()

callback when working against new servers.

ra_serf: Add new bucket that wraps a readable svn_stream_t.

This new bucket would be required to make ra_serf stream svndiff deltas

without creating temporary files. (We'll take an svn_txdelta_stream_t,

convert it to a generic svn_stream_t with svn_txdelta_to_svndiff_stream(),

create a bucket around that stream, and use it as the PUT request body.

This change introduces adds the necessary bucket implementation.)

* subversion/include/svn_error_codes.h

(SVN_ERR_RA_SERF_STREAM_BUCKET_READ_ERROR): New.

* subversion/libsvn_ra_serf/ra_serf.h

(svn_ra_serf__stream_bucket_errfunc_t): Declare new callback type.

(svn_ra_serf__create_stream_bucket): Declare new function.

* subversion/libsvn_ra_serf/stream_bucket.c: New file.

(stream_bucket_ctx_t): New.

(svn_ra_serf__create_stream_bucket): New, creates the bucket and

initializes a databuf that would be used in its implementation.

Remember the passed-in errfunc callback to allow reporting extended

error information to the caller in the form of an svn_error_t.

(stream_bucket_read, stream_bucket_readline, stream_bucket_peek):

New bucket vtable members that forward everything to the databuf.

(stream_reader): New. This is the core function of the new bucket that

reads data from the svn_stream_t and possibly forwards the errors

using the stored errfunc callback.

Add new svn_txdelta_to_svndiff_stream() API.

This API turns a given svn_txdelta_stream_t delta stream into a readable

svn_stream_t containing svndiff. It would be required to make ra_serf

stream svndiff deltas without creating temporary files.

* subversion/include/svn_delta.h

(svn_txdelta_to_svndiff_stream): Declare new function.

* subversion/libsvn_delta/svndiff.c

(svndiff_stream_baton_t): New.

(svndiff_stream_write_fn): New, writes svndiff to a temporary buffer.

(svndiff_stream_read_fn): New, passes the buffered data to the caller and

refills the internal buffer if necessary.

(svn_txdelta_to_svndiff_stream): Implement this new function. Set up

pull- and push-streams. Install the read and write stream handlers, and

return the pull stream to the caller.

* subversion/tests/libsvn_delta/random-test.c

(do_random_txdelta_to_svndiff_stream_test): Core of the new test, similar

to do_random_test().

(random_txdelta_to_svndiff_stream_test): Run new test.

(test_funcs): Add new test.

* tools/buildbot/slaves/svn-x64-macosx/svnbuild.sh:

When building svn >= 1.10, build against the internal copy of lz4.

Build against the system LZ4 library by default instead of the

internal copy of the LZ4 code. Add --with-lz4=PREFIX|internal

to allow the user to specify either where to find LZ4 or that

the internal copy be used. The Windows build will continue to

use the internal LZ4 code.

* build/ac-macros/lz4.m4: New.

* aclocal.m4: Include lz4.m4.

* configure.ac: Call SVN_LZ4.

* Makefile.in

(SVN_LZ4_LIBS, SVN_LZ4_INCLUDES): New.

* build.conf

(libsvn_subr): Add lz4 to link.

(lz4): New.

* subversion/libsvn_subr/lz4/lz4.h: Rename to ...

* subversion/libsvn_subr/lz4/lz4internal.h: ... this, include

svn_private_config.h, protect with SVN_INTERNAL_LZ4.

* subversion/libsvn_subr/lz4/lz4.c: Include svn_private_config.h and

lz4internal.h, protect with SVN_INTERNAL_LZ4.

* subversion/libsvn_subr/compress.c: Include system lz4.h or lz4internal.h.

* subversion/svn_private_config.hw: Define SVN_INTERNAL_LZ4 to

enable internal copy of LZ4.

On the 'shelve-checkpoint3' branch: Error out appropriately when

checkpointing is or is not initialized.

On the 'shelve-checkpoint3' branch: Destroy the checkpoints repo

at the end of the 'checkpoint finish' command.

On the 'shelve-checkpoint3' branch: Print more verbose hints about

what it is doing.

* docs/release-notes/1.10.html

(#new-feature-compatibility-table): Split the lz4 entry to match

the implementation.

* site/publish/docs/release-notes/1.10.html

(new-feature-compatibility-table): Port the note about the program being

both the server and the client over file:// from 1.9 release notes.

* docs/community-guide/releasing.part.html

(#releasing-update-website): Spell out another manual step.

* index.html, news.html: Link to 1.10.0-alpha3 announcement.
Release 1.10.0-alpha3.
* docs/community-guide/releasing.part.html

(#releasing-update-website): Document 'release.py write-downloads'.

* docs/community-guide/releasing.part.html

(#prerelease-caveats): Explain the FSFS f5 example better.