Checkout Tools
  • last updated 5 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Fix an issue with the readline implementation for file streams that could

cause excessive memory usage for inputs containing one or multiple \0 bytes.

This is the likely cause of the OOM reported in

https://lists.apache.org/thread.html/c96eb5618ac0bf6e083345e0fdcdcf834e30913f26eabe6ada7bab62@%3Cusers.subversion.apache.org%3E

(I think that the problem itself is a regression introduced by me in 1.10.)

Note: one thing I noticed while preparing the fix is that our `readline_fn`

functions for different streams have inconsistent behavior if the input data

contains \0 bytes. More specifically, they may return different `line` values,

that may either be truncated at \0 or actually contain the whole data between

EOLs, including \0 bytes. For now, this patch only fixes the excessive memory

usage problem, and I noted this related problem in the test and left it for

future work.

* subversion/libsvn_subr/stream.c

(readline_apr_lf, readline_apr_generic): Reallocate the buffer based on its

current size, instead of calculating the new size based on the already

prealloc'd size. There are no actual benefits in reallocating based on

`blocksize`, and in the described case with \0 bytes doing so also backfires

and may cause excessive allocations due to the actual size of the string

being less than we expect it to. A degenerate case of the erroneous

behavior is ...

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_readline_file_nul): ...exploited in this new test.

(test_funcs): Run new test.

* subversion/tests/libsvn_subr

(): Adjust svn:ignore.

  1. … 2 more files in changeset.
Fix the excessive amount of read and seek syscalls in various fsfs operations

that read from disk.

The idea behind the fix is that we provide a special svn_stream_readline()

implementation for APR file-based streams that handles a common case of

LF ("\n") end-of-line sequence with apr_file_gets().

For example, this patch turns the following sequence ...

open("/repo/db/revs/22.pack/manifest", O_RDONLY|O_CLOEXEC) = 4

read(4, "0\n3304\n7139\n12288\n13981\n37151\n59"..., 4096) = 4096

read(4, "5140\n20998974\n21002732\n21004670\n"..., 4096) = 4096

lseek(4, 4020, SEEK_SET) = 4020

read(4, "18102627\n18139220\n18141921\n18147"..., 4096) = 4096

read(4, "2535219\n52537641\n52550290\n525536"..., 4096) = 584

lseek(4, 8043, SEEK_SET) = 8043

read(4, "51452518\n51454318\n51456275\n51457"..., 4096) = 657

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

read(4, "", 4096) = 0

close(4) = 0

...into this:

open("/repo/db/revs/22.pack/manifest", O_RDONLY|O_CLOEXEC) = 4

read(4, "0\n3304\n7139\n12288\n13981\n37151\n59"..., 4096) = 4096

read(4, "5140\n20998974\n21002732\n21004670\n"..., 4096) = 4096

read(4, "131\n52730772\n52744939\n52748273\n5"..., 4096) = 508

read(4, "", 4096) = 0

close(4)

* subversion/libsvn_subr/stream.c

(readline_apr_lf): New.

(readline_apr_generic): New, factored out from ...

(readline_apr_handler): ...here. Call the apr_file_gets()-based routine

when we're looking for an LF end-of-line sequence to avoid unnecessary

syscalls.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_readline_file): New helper for ...

(test_stream_readline_file_lf, test_stream_readline_file_crlf): ...these

new tests that cover svn_stream_readline() behavior for LF and CRLF

terminated lines.

(test_funcs): Add new tests.

  1. … 1 more file in changeset.
Follow-up to r1755486: Rename svn_stream_checksum() to

svn_stream_contents_checksum().

Suggested by: danielsh

* subversion/include/svn_io.h

* subversion/libsvn_subr/stream.c

(svn_stream_checksum): Rename to svn_stream_contents_checksum().

(compute_stream_checksum): Update docstring.

* subversion/libsvn_fs/fs-loader.c

* subversion/libsvn_subr/io.c

* subversion/tests/libsvn_fs/fs-test.c

* subversion/tests/libsvn_subr/stream-test.c

(svn_fs_file_checksum, svn_io_file_checksum2, get_file_checksum,

test_stream_checksum): Adapt callers.

  1. … 5 more files in changeset.
Introduce svn_stream_checksum() function to calculate checksum of specified

stream contents. Use new API where it makes sense.

* subversion/include/svn_io.h

(svn_stream_checksum): New.

* subversion/libsvn_subr/stream.c

(compute_stream_checksum): New. Helper for svn_stream_checksum().

(svn_stream_checksum): New.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_checksum): New. Simple test for svn_stream_checksum().

(test_funcs): Add test_stream_checksum to test list.

* subversion/libsvn_fs/fs-loader.c

* subversion/libsvn_repos/config_pool.c

* subversion/libsvn_subr/io.c

* subversion/tests/libsvn_fs/fs-test.c

(svn_fs_file_checksum, auto_parse, svn_io_file_checksum2,

get_file_checksum): Use svn_stream_checksum() instead of

svn_stream_checksummed2(READ_ALL=TRUE) + svn_stream_close().

  1. … 6 more files in changeset.
Implement svn_base64_encode2() with BREAK_LINES flag to control whether

insert newlines periodically or not. The old svn_base64_encode() was

inserting newlines unconditionally.

* subversion/include/svn_base64.h

(svn_base64_encode2): New function declaration.

(svn_base64_encode): Deprecate.

* subversion/libsvn_subr/base64.c

(encode_baton): Add BREAK_LINES field.

(encode_data, finish_encoding_data): Use EB->BREAK_LINES flag.

(svn_base64_encode2): Revv from svn_base64_encode(). Add BREAK_LINES.

argument and save it in ENCODE_BATON.

* subversion/libsvn_subr/deprecated.c

(svn_base64_encode): Call svn_base64_encode2() with BREAK_LINES=TRUE.

* subversion/mod_dav_svn/util.c

(dav_svn__make_base64_output_stream): Use svn_base64_encode2() with

BREAK_LINES=TRUE.

* subversion/tests/libsvn_delta/svndiff-test.c

(main): Use svn_base64_encode2() with BREAK_LINES=TRUE.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_base64, test_stream_base64_2): Test svn_base64_encode2() with

BREAK_LINES=TRUE and BREAK_LINES=FALSE.

  1. … 5 more files in changeset.
Remove the now unused svn_stream_wrap_buffered_read() functionality.

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Drop.

* subversion/libsvn_subr/stream.c

(buffering_stream_wrapper_baton,

read_handler_buffering_wrapper,

data_available_handler_buffering_wrapper,

is_buffered_handler_buffering_wrapper,

svn_stream_wrap_buffered_read): Drop.

* subversion/tests/libsvn_subr/stream-test.c

(stream_baton_t,

read_handler,

data_available_handler,

create_test_read_stream,

expect_line_content,

test_stream_buffered_wrapper): Drop.

(test_funcs): Unregister test.

  1. … 2 more files in changeset.
[Combines r1698359 and r170078 into a single commit for better review.]

Introduce a stream wrapper object that adds buffering support to any

readable stream. Use it on the stdin streams in our CL tools.

As it turns out, parsing data from a stdin byte-by-byte incurs a

massive overhead of 100% internal and 300% system load over a buffered

stream. 'svnadmin load-revprops' sees a 5 times speedup if all data

is in OS disc caches. This is a realistic assumption in a "final sync

and switch over to new repository" scenario.

The other 2 uses of stdin either have less data to process (svnfsfs

load-index) or parse only a small fraction of the stream (svnadmin load).

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Declare the new stream constructor API.

* subversion/libsvn_subr/stream.c

(read_handler_buffering_wrapper,

data_available_handler_buffering_wrapper,

is_buffered_handler_buffering_wrapper): Internal logic of the new

stream object.

(svn_stream_wrap_buffered_read): New constructor implementation.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Wrap the stdin stream.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Same.

* subversion/tests/libsvn_subr/stream-test.c

(struct stream_baton_t,

read_handler,

data_available_handler,

create_test_read_stream): New configurable test read stream.

(expect_line_content,

test_stream_buffered_wrapper): New test for the new wrapper stream.

(test_funcs): Register the new test.

  1. … 4 more files in changeset.
Revert r1698359 and r1700789 for rehashing.
  1. … 4 more files in changeset.
[Reverted in r1700797 and re-applied as part of r1700799.]

Remove support for mark & seek from "buffered read" stream wrapper as it

was deemed to difficult to handle. See also here:

http://mail-archives.apache.org/mod_mbox/subversion-dev/201509.mbox/%3CCAP_GPNjwhjD1Ds5%2ByYKiqr2NwzbhmnGq%3DqtB8jowBUJty7_Z3Q%40mail.gmail.com%3E

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Remove all references to mark & seek

other than saying we don't support them.

* subversion/libsvn_subr/stream.c

(buffering_stream_wrapper_baton): Remove all elements used for mark & seek.

(buffering_stream_wrapper_mark): Drop.

(read_handler_buffering_wrapper): Simplify as we always exhaust the buffer

and can completely discard it now.

(decrement_mark_count,

mark_handler_buffering_wrapper,

seek_handler_buffering_wrapper,

assert_zero_mark_count): Drop.

(svn_stream_wrap_buffered_read): Update. Allocate the buffer only one and

do that here.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Update API caller.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Update API caller.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_buffered_wrapper): Update API caller.

  1. … 4 more files in changeset.
[Reverted in r1700797 and re-applied as part of r1700799.]

Introduce a stream wrapper object that adds mark/seek support to any

readable stream. Use it on the stdin streams in our CL tools.

As it turns out, parsing data from a stdin byte-by-byte incurs a

massive overhead of 100% internal and 300% system load over a buffered

stream. 'svnadmin load-revprops' sees a 5 times speedup if all data

is in OS disc caches. This is a realistic assumption in a "final sync

and switch over to new repository" scenario.

The other 2 uses of stdin either have less data to process (svnfsfs

load-index) or parse only a small fraction of the stream (svnadmin load).

To avoid any memory usage issue due to the added buffering, svnadmin

load will not use the stream wrapper - the loader might clean up some

of the pools only once per revision.

* subversion/include/svn_io.h

(svn_stream_wrap_buffered_read): Declare the new stream constructor API.

* subversion/libsvn_subr/stream.c

(buffering_stream_wrapper_baton,

buffering_stream_wrapper_mark): New data structures describing the

wrapper stream and marker states.

(read_handler_buffering_wrapper,

decrement_mark_count,

mark_handler_buffering_wrapper,

seek_handler_buffering_wrapper,

data_available_handler_buffering_wrapper,

is_buffered_handler_buffering_wrapper,

assert_zero_mark_count): Internal logic of the new stream object.

(svn_stream_wrap_buffered_read): New constructor implementation.

* subversion/svnadmin/svnadmin.c

(subcommand_load_revprops): Wrap the stdin stream.

* subversion/svnfsfs/load-index-cmd.c

(subcommand__load_index): Same.

* subversion/tests/libsvn_subr/stream-test.c

(struct stream_baton_t,

read_handler,

data_available_handler,

create_test_read_stream): New configurable test read stream.

(expect_line_content,

test_stream_buffered_wrapper): New test for the new wrapper stream.

(test_funcs): Register the new test.

  1. … 4 more files in changeset.
Fix svn_stream_compressed() for streams without partial read support.

* subversion/libsvn_subr/stream.c

(read_helper_gz): Use svn_stream_read_full() to read data from substream.

* subversion/tests/libsvn_subr/stream-test.c

(empty_read_full_fn): Helper for test_stream_compressed_read_full() test.

(test_stream_compressed_read_full): New test.

(test_funcs): Add test test_stream_compressed_read_full() to test list.

  1. … 1 more file in changeset.
Make the C tests work with the --enable-optimize configure option.

Because the main() function used to be defined in libsvn_test,

the -flto option (which performs rather aggressive link-time

optimizations) would cause the linker to remove all "unreferenced"

static functions ... which includes all the actual tests.

Which this change, main() is now defined in each test driver.

* subversion/tests/svn_test.h

(test_funcs, svn_test_max_threads): Remove external declarations.

(svn_test_main): New prototype for the (existing-but-renamed) test driver.

(SVN_TEST_MAIN): Bolierplate macro that generates a main() implementation.

* subversion/tests/svn_test_main.c

(svn_test_main): Renamed from 'main', and takes two extra parameters,

'max_threads' and 'test_funcs'. Propagates test_funcs to get_array_size,

do_test_num and do_tests_concurrently.

(get_array_size, do_test_num,

test_params_t, test_thread, do_tests_concurrently): Require a test_funcs

parameter. All uses updated.

* subversion/tests/libsvn_client/client-test.c,

subversion/tests/libsvn_client/mtcc-test.c,

subversion/tests/libsvn_delta/random-test.c,

subversion/tests/libsvn_delta/window-test.c,

subversion/tests/libsvn_diff/diff-diff3-test.c,

subversion/tests/libsvn_diff/parse-diff-test.c,

subversion/tests/libsvn_fs/fs-test.c,

subversion/tests/libsvn_fs/locks-test.c,

subversion/tests/libsvn_fs_base/

subversion/tests/libsvn_fs_base/changes-test.c,

subversion/tests/libsvn_fs_base/fs-base-test.c,

subversion/tests/libsvn_fs_base/strings-reps-test.c,

subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c,

subversion/tests/libsvn_fs_x/fs-x-pack-test.c,

subversion/tests/libsvn_fs_x/string-table-test.c,

subversion/tests/libsvn_ra/ra-test.c,

subversion/tests/libsvn_ra_local/ra-local-test.c,

subversion/tests/libsvn_repos/repos-test.c,

subversion/tests/libsvn_subr/auth-test.c,

subversion/tests/libsvn_subr/cache-test.c,

subversion/tests/libsvn_subr/checksum-test.c,

subversion/tests/libsvn_subr/compat-test.c,

subversion/tests/libsvn_subr/config-test.c,

subversion/tests/libsvn_subr/crypto-test.c,

subversion/tests/libsvn_subr/dirent_uri-test.c,

subversion/tests/libsvn_subr/error-code-test.c,

subversion/tests/libsvn_subr/error-test.c,

subversion/tests/libsvn_subr/hashdump-test.c,

subversion/tests/libsvn_subr/io-test.c,

subversion/tests/libsvn_subr/mergeinfo-test.c,

subversion/tests/libsvn_subr/named_atomic-test.c,

subversion/tests/libsvn_subr/opt-test.c,

subversion/tests/libsvn_subr/packed-data-test.c,

subversion/tests/libsvn_subr/path-test.c,

subversion/tests/libsvn_subr/prefix-string-test.c,

subversion/tests/libsvn_subr/priority-queue-test.c,

subversion/tests/libsvn_subr/revision-test.c,

subversion/tests/libsvn_subr/root-pools-test.c,

subversion/tests/libsvn_subr/skel-test.c,

subversion/tests/libsvn_subr/spillbuf-test.c,

subversion/tests/libsvn_subr/sqlite-test.c,

subversion/tests/libsvn_subr/stream-test.c,

subversion/tests/libsvn_subr/string-test.c,

subversion/tests/libsvn_subr/subst_translate-test.c,

subversion/tests/libsvn_subr/time-test.c,

subversion/tests/libsvn_subr/translate-test.c,

subversion/tests/libsvn_subr/utf-test.c,

subversion/tests/libsvn_wc/conflict-data-test.c,

subversion/tests/libsvn_wc/db-test.c,

subversion/tests/libsvn_wc/entries-compat.c,

subversion/tests/libsvn_wc/op-depth-test.c,

subversion/tests/libsvn_wc/pristine-store-test.c,

subversion/tests/libsvn_wc/wc-queries-test.c,

subversion/tests/libsvn_wc/wc-test.c:

(max_threads): Renamed from svn_test_max_threads and made static.

(test_funcs): Made static.

(SVN_TEST_MAIN): Expanded boilerplate main() function.

  1. … 54 more files in changeset.
Following up on r1561688, switch almost every caller of svn_stream_read() to

its successor svn_stream_read_full(). Do this to resolve many deprecation

warnings (and to avoid somebody else accidentally doing the invalid conversion

to svn_stream_read2()).

There are no functional changes as svn_stream_read() is a direct call to

svn_stream_read_full().

* subversion/libsvn_delta/svndiff.c

(read_one_byte): Update caller.

* subversion/libsvn_delta/text_delta.c

(txdelta_next_window,

tpush_write_handler,

apply_window,

svn_txdelta_send_stream): Update caller.

* subversion/libsvn_fs_base/lock.c

(svn_fs_base__get_locks): Update caller.

* subversion/libsvn_fs_fs/recovery.c

(read_handler_recover): Update caller.

* subversion/libsvn_fs_x/cached_data.c

(get_dir_contents): Update caller.

* subversion/libsvn_fs_x/pack.c

(write_reps_containers): Update caller.

* subversion/libsvn_repos/load.c

(read_key_or_val,

parse_text_block,

svn_repos_parse_dumpstream3): Update caller.

* subversion/libsvn_subr/config_file.c

(parser_getc): Update caller.

* subversion/libsvn_subr/hash.c

(svn_hash__read_entry):

* subversion/libsvn_subr/packed_data.c

(read_stream_uint,

read_stream_data,

svn_packed__data_read): Update caller.

* subversion/libsvn_subr/subst.c

(translated_stream_read,

read_handler_special): Update caller.

* subversion/libsvn_wc/adm_crawler.c

(read_handler_copy): Update caller.

* subversion/mod_dav_svn/repos.c

(deliver): Update caller.

* subversion/svnserve/serve.c

(get_file): Update caller.

* subversion/tests/libsvn_subr/spillbuf-test.c

(test_spillbuf_stream): Update caller.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_from_string,

test_stream_compressed,

test_stream_seek_stringbuf,

test_stream_seek_translated,

test_stream_compressed_empty_file): Update caller.

* subversion/tests/libsvn_wc/pristine-store-test.c

(pristine_delete_while_open): Update caller.

* subversion/tests/svn_test_fs.c

(svn_test__stream_to_string): Update caller.

* tools/server-side/fsfs-stats.c

(read_windows): Update caller.

  1. … 18 more files in changeset.
Add "--parallel" option to our C tests.

Besides shaving off a few seconds off our total test execution time,

parallel execution also stresses our code and the tests themselves

harder. It uncovers things like improperly separated working sets

that make post-failure analysis harder.

Parallel mode is only available with APR 1.3+ and threads enabled.

The option will simply be ignored in other configurations. Further

limitations are cleanups being delayed to after all tests completed

and segfaults not being intercepted.

Some tests can't be executed in parallel. Therefore, we introduce

svn_test_max_threads a simple upper limit to the number of threads

supported that each test must define alongside the test_funcs array.

This patch sets it to "1" in many cases because the tests are so

small that multi-threaded execution is simply not worth it.

* build/run_tests.py

(TestHarness._run_c_test): Pass --parallel option to the individual

test applications.

* subversion/tests/svn_test.h

(svn_test_max_threads): Declare new external setting to be set by

every test application.

* subversion/tests/svn_test_main.c

(parallel): New command line option presence flag.

(parallel_opt,

cl_options): Declare the new "--parallel" option.

(svn_test_add_dir_cleanup): Synchronize shared pool access since

this might be called from multiple

threads at the same time.

(log_results): Result logger function factored out from do_test_num

to be shared between serialized and parallel test

execution code.

(do_test_num): Use that new function.

(test_params_t,

test_thread,

do_tests_concurrently): Threads, their parameter and the thread

starter for thread-pooled test execution.

(main): Handle the new option.

Call concurrent execution when in PARALLEL mode.

* subversion/tests/libsvn_delta/random-test.c

subversion/tests/libsvn_subr/named_atomic-test.c

(svn_test_max_threads): These tests don't support concurrent execution.

* subversion/tests/libsvn_delta/random-test.c

subversion/tests/libsvn_delta/window-test.c

subversion/tests/libsvn_diff/parse-diff-test.c

subversion/tests/libsvn_ra_local/ra-local-test.c

subversion/tests/libsvn_ra/ra-test.c

subversion/tests/libsvn_subr/auth-test.c

subversion/tests/libsvn_subr/cache-test.c

subversion/tests/libsvn_subr/checksum-test.c

subversion/tests/libsvn_subr/compat-test.c

subversion/tests/libsvn_subr/config-test.c

subversion/tests/libsvn_subr/crypto-test.c

subversion/tests/libsvn_subr/dirent_uri-test.c

subversion/tests/libsvn_subr/error-code-test.c

subversion/tests/libsvn_subr/error-test.c

subversion/tests/libsvn_subr/hashdump-test.c

subversion/tests/libsvn_subr/mergeinfo-test.c

subversion/tests/libsvn_subr/named_atomic-test.c

subversion/tests/libsvn_subr/opt-test.c

subversion/tests/libsvn_subr/packed-data-test.c

subversion/tests/libsvn_subr/path-test.c

subversion/tests/libsvn_subr/prefix-string-test.c

subversion/tests/libsvn_subr/priority-queue-test.c

subversion/tests/libsvn_subr/revision-test.c

subversion/tests/libsvn_subr/root-pools-test.c

subversion/tests/libsvn_subr/skel-test.c

subversion/tests/libsvn_subr/spillbuf-test.c

subversion/tests/libsvn_subr/sqlite-test.c

subversion/tests/libsvn_subr/stream-test.c

subversion/tests/libsvn_subr/string-test.c

subversion/tests/libsvn_subr/subst_translate-test.c

subversion/tests/libsvn_subr/time-test.c

subversion/tests/libsvn_subr/utf-test.c

subversion/tests/libsvn_wc/conflict-data-test.c

subversion/tests/libsvn_wc/wc-queries-test.c:

(svn_test_max_threads): These are too small to benefit from multi-threading.

* subversion/tests/libsvn_client/client-test.c

subversion/tests/libsvn_diff/diff-diff3-test.c

subversion/tests/libsvn_fs_base/changes-test.c

subversion/tests/libsvn_fs_base/fs-base-test.c

subversion/tests/libsvn_fs_base/strings-reps-test.c

subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

subversion/tests/libsvn_fs/fs-test.c

subversion/tests/libsvn_fs/locks-test.c

subversion/tests/libsvn_fs_x/fs-x-pack-test.c

subversion/tests/libsvn_fs_x/string-table-test.c

subversion/tests/libsvn_repos/repos-test.c

subversion/tests/libsvn_subr/io-test.c

subversion/tests/libsvn_subr/translate-test.c

subversion/tests/libsvn_wc/db-test.c

subversion/tests/libsvn_wc/op-depth-test.c

subversion/tests/libsvn_wc/wc-test.c

(svn_test_max_threads): Set concurrency limit to whatever various test

runs suggested as the most efficient value.

* subversion/tests/libsvn_wc/entries-compat.c

subversion/tests/libsvn_wc/pristine-store-test.c

(svn_test_max_threads): Set concurrency limit to "unbounded".

  1. … 54 more files in changeset.
Introduce svn_stringbuf_from_stream() as symmetric counterpart to

svn_stream_from_stringbuf() and use it in one place.

* subversion/include/svn_io.h

(svn_stringbuf_from_stream): declare new public API function

* subversion/libsvn_subr/stream.c

(svn_stringbuf_from_stream): implement new API function

* subversion/libsvn_fs_fs/cached_data.c

(get_dir_contents): use the new API

* develop/trunk/subversion/tests/libsvn_subr/stream-test.c

(test_stringbuf_from_stream): new test case for new API

(test_funcs): register new test case

  1. … 3 more files in changeset.
Coding style patch: in logical expressions, instead of comparing

with arithmetic constants use boolean operations. I.e. replace

<expr> == TRUE with <expr> and

<expr> == FALSE with !<expr>

* subversion/libsvn_delta/svndiff.c

(send_simple_insertion_window,

window_handler): change condition style

* subversion/libsvn_fs_base/fs.c

(svn_fs_base__clean_logs): ditto

* subversion/libsvn_fs_fs/fs_fs.c

(representation_string,

create_rep_state_body,

build_rep_list): ditto

* subversion/libsvn_ra_serf/locks.c

(handle_lock): ditto

* subversion/libsvn_ra_serf/property.c

(create_propfind_body): same here,

* subversion/libsvn_ra_serf/update.c

(cancel_fetch,

handle_fetch,

end_report,

finish_report): here,

* subversion/libsvn_ra_serf/util.c

(svn_ra_serf__handle_xml_parser): here,

* subversion/libsvn_ra_svn/client.c

(optbool_to_tristate): here,

* subversion/libsvn_repos/authz.c

(authz_parse_section): and here.

* subversion/libsvn_repos/fs-wrap.c

(svn_repos__validate_prop): here as well

* subversion/libsvn_subr/cmdline.c

(svn_cmdline_create_auth_baton): and here

* subversion/libsvn_subr/deprecated.c

(print_command_info): here

* subversion/libsvn_subr/opt.c

(print_command_info2): more of the same

* subversion/libsvn_subr/subst.c

(translate_chunk): ditto

* subversion/libsvn_subr/win32_crashrpt.c

(write_var_values,

svn__unhandled_exception_filter): ditto

* subversion/svn/svn.c

(sub_main): ditto

* subversion/svnsync/svnsync.c

(filter_props): and more of the

* subversion/tests/libsvn_subr/stream-test.c

(test_readonly): same thing

* subversion/tests/libsvn_subr/string-test.c

(test21): ditto

* subversion/tests/libsvn_subr/subst_translate-test.c

(test_svn_subst_translate_string2_null_encode): here

* subversion/tests/libsvn_wc/db-test.c

(test_getting_info,

test_working_info): and here

* tools/client-side/svn-bench/svn-bench.c

(sub_main): and finally here

  1. … 21 more files in changeset.
In this revision a mass spelling correction OCCURRED (not occured).

* subversion/bindings/javahl/native/Path.cpp,

subversion/bindings/javahl/native/Path.h

(m_error_occured): Rename to 'm_error_occurred'.

(error_occured): Rename to 'error_occurred'.

* subversion/bindings/javahl/native/Targets.cpp,

subversion/bindings/javahl/native/Targets.h

(Targets): Same.

* subversion/bindings/javahl/native/SVNClient.cpp

Track the renames.

* subversion/bindings/swig/perl/native/Repos.pm

(get_logs): Fix spelling in a doc string.

* subversion/libsvn_subr/cache_config.c

(svn_cache__get_global_membuffer_cache): Fix spelling in a comment.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_base64_2): Fix spelling in a doc string.

* tools/server-side/svn-backup-dumps.py

(__main__): Fix spelling in printed output.

* contrib/client-side/svnmerge/svnmerge_test.py

(TestCase_TestRepo.testMergeAndRollbackEmptyRevisionRange): Fix spelling

in a doc string.

  1. … 9 more files in changeset.
Silence GCC warnings when assigning to chars (-Wconversion).

In all these cases, the value may never exceed the [-128,127]

range for char.

* subversion/libsvn_delta/svndiff.c

(window_handler): explicitly cast to char

* subversion/libsvn_fs_fs/key-gen.c

(svn_fs_fs__next_key): prevent conversion to int

* subversion/libsvn_fs_fs/temp_serializer.c

(encode_number): explicitly cast to char

* subversion/libsvn_ra_svn/marshal.c

(svn_ra_svn_write_string, svn_ra_svn_write_cstring): ditto

* subversion/libsvn_subr/base64.c

(decode_group): ditto

* subversion/libsvn_subr/checksum.c

(svn_checksum_parse_hex): same here

* subversion/libsvn_subr/dirent_uri.c

(canonicalize_to_lower, canonicalize_to_upper): here

* subversion/mod_dav_svn/repos.c

(get_entry): and here

* subversion/tests/libsvn_repos/repos-test.c

(prop_validation): char

* subversion/tests/libsvn_subr/stream-test.c

(generate_test_bytes): explicitly cast to char

  1. … 9 more files in changeset.
* subversion/tests/libsvn_subr/stream-test.c

(test_stream_base64_2): Add explanation.

Fix a memory allocation problem in the base64 code that caused an

occasional write beyond the end of a buffer during decoding.

* subversion/libsvn_subr/svn_base64.c

(decode_bytes): Allocate enough space.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_base64_2): New.

(test_funcs): Add new test.

  1. … 1 more file in changeset.
Reintegrate integrate-string-improvements branch into /trunk.
  1. … 83 more files in changeset.
Simplify the semantics of svn_stream_skip() by not promising to report the

actual length skipped if an error (including EOF) occurs. Thus eliminate a

bug in reporting that length in skip_handler_apr().

See email by Stefan Fuhrmann on 2011-06-02, "svn commit: r1072519 - Add

svn_stream_skip(), svn_stream_buffered(), svn_stream_supports_mark()",

<http://svn.haxx.se/dev/archive-2011-06/0038.shtml>.

* subversion/include/svn_io.h

(svn_skip_fn_t, svn_stream_skip): Change the '*len' in/out parameter to

'len', input-only.

* subversion/libsvn_subr/stream.c

(svn_stream_skip, skip_handler_disown, skip_handler_md5,

skip_handler_stringbuf, skip_handler_string): Same.

(skip_default_handler, skip_handler_apr): Same, and thus simplify.

(stream_readline_chunky): Adjust a call.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_seek_file, test_stream_seek_stringbuf,

test_stream_seek_translated): Adjust calls.

  1. … 2 more files in changeset.
Add simple test for base64 encode/streams.

* subversion/tests/libsvn_subr/stream-test.c

(): Include svn_base64.h.

(test_stream_base64): New.

(test_funcs): Update test list.

* subversion/libsvn_subr/stream.c

(read_handler_gz): Exit early if the underlying stream ran out instead

of trying to decompress more data. Based on an initial diff by Ivan.

* subversion/tests/libsvn_subr/stream-test.c

(test_funcs): Remove XFAIL marker from test_stream_compressed_empty_file.

  1. … 1 more file in changeset.
* subversion/tests/libsvn_subr/stream-test.c

(test_stream_compressed_empty_file): Eliminate TEST_BUF_SIZE macro and

reduce buffer size to one byte since we do not expect any data.

Follow-up to r1125814:

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_compressed_empty_file): Remove unused variables.

Add an XFAIL test for a problem discovered by Ivan.

A compressed stream reading from an empty file results in a zlib error:

subversion/tests/libsvn_subr/stream-test.c:546: (apr_err=12)

subversion/libsvn_subr/stream.c:1092: (apr_err=12)

svn_tests: E000012: zlib (inflate): buffer error

The problem seems to be specific to streams backed by APR files.

Manual testing shows that reading from empty stringbufs works fine.

* subversion/tests/libsvn_subr/stream-test.c

(test_stream_compressed_empty_file, test_funcs): New test.

* subversion/tests/libsvn_sutr/stream-test.c

(test_stream_compressed): Fix a typo in macro name passed to #undef.

Fix the read-only detection in svn_io__is_finfo_read_only().

* subversion/libsvn_subr/io.c

(svn_io__is_finfo_read_only): Check for APR_FREADONLY.

* subversion/tests/libsvn_subr/stream-test.c

(includes): Add private/svn_io_private.h.

(test_readonly): New function.

(test_funcs): Add test_readonly.

  1. … 1 more file in changeset.
Merge all changes (r1068695 - r1072516) from the

integrate-stream-api-extensions.

These patches add svn_stream_skip(), svn_stream_buffered()

and svn_stream_supports_mark().

  1. … 4 more files in changeset.