Checkout Tools
  • last updated 8 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Cherry-pick merge r1620599 from branches/revprop-caching-ng without conflicts.

This removes the named_atomics code.

  1. … 6 more files in changeset.
Make the C tests work with the --enable-optimize configure option.

Because the main() function used to be defined in libsvn_test,

the -flto option (which performs rather aggressive link-time

optimizations) would cause the linker to remove all "unreferenced"

static functions ... which includes all the actual tests.

Which this change, main() is now defined in each test driver.

* subversion/tests/svn_test.h

(test_funcs, svn_test_max_threads): Remove external declarations.

(svn_test_main): New prototype for the (existing-but-renamed) test driver.

(SVN_TEST_MAIN): Bolierplate macro that generates a main() implementation.

* subversion/tests/svn_test_main.c

(svn_test_main): Renamed from 'main', and takes two extra parameters,

'max_threads' and 'test_funcs'. Propagates test_funcs to get_array_size,

do_test_num and do_tests_concurrently.

(get_array_size, do_test_num,

test_params_t, test_thread, do_tests_concurrently): Require a test_funcs

parameter. All uses updated.

* subversion/tests/libsvn_client/client-test.c,

subversion/tests/libsvn_client/mtcc-test.c,

subversion/tests/libsvn_delta/random-test.c,

subversion/tests/libsvn_delta/window-test.c,

subversion/tests/libsvn_diff/diff-diff3-test.c,

subversion/tests/libsvn_diff/parse-diff-test.c,

subversion/tests/libsvn_fs/fs-test.c,

subversion/tests/libsvn_fs/locks-test.c,

subversion/tests/libsvn_fs_base/

subversion/tests/libsvn_fs_base/changes-test.c,

subversion/tests/libsvn_fs_base/fs-base-test.c,

subversion/tests/libsvn_fs_base/strings-reps-test.c,

subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c,

subversion/tests/libsvn_fs_x/fs-x-pack-test.c,

subversion/tests/libsvn_fs_x/string-table-test.c,

subversion/tests/libsvn_ra/ra-test.c,

subversion/tests/libsvn_ra_local/ra-local-test.c,

subversion/tests/libsvn_repos/repos-test.c,

subversion/tests/libsvn_subr/auth-test.c,

subversion/tests/libsvn_subr/cache-test.c,

subversion/tests/libsvn_subr/checksum-test.c,

subversion/tests/libsvn_subr/compat-test.c,

subversion/tests/libsvn_subr/config-test.c,

subversion/tests/libsvn_subr/crypto-test.c,

subversion/tests/libsvn_subr/dirent_uri-test.c,

subversion/tests/libsvn_subr/error-code-test.c,

subversion/tests/libsvn_subr/error-test.c,

subversion/tests/libsvn_subr/hashdump-test.c,

subversion/tests/libsvn_subr/io-test.c,

subversion/tests/libsvn_subr/mergeinfo-test.c,

subversion/tests/libsvn_subr/named_atomic-test.c,

subversion/tests/libsvn_subr/opt-test.c,

subversion/tests/libsvn_subr/packed-data-test.c,

subversion/tests/libsvn_subr/path-test.c,

subversion/tests/libsvn_subr/prefix-string-test.c,

subversion/tests/libsvn_subr/priority-queue-test.c,

subversion/tests/libsvn_subr/revision-test.c,

subversion/tests/libsvn_subr/root-pools-test.c,

subversion/tests/libsvn_subr/skel-test.c,

subversion/tests/libsvn_subr/spillbuf-test.c,

subversion/tests/libsvn_subr/sqlite-test.c,

subversion/tests/libsvn_subr/stream-test.c,

subversion/tests/libsvn_subr/string-test.c,

subversion/tests/libsvn_subr/subst_translate-test.c,

subversion/tests/libsvn_subr/time-test.c,

subversion/tests/libsvn_subr/translate-test.c,

subversion/tests/libsvn_subr/utf-test.c,

subversion/tests/libsvn_wc/conflict-data-test.c,

subversion/tests/libsvn_wc/db-test.c,

subversion/tests/libsvn_wc/entries-compat.c,

subversion/tests/libsvn_wc/op-depth-test.c,

subversion/tests/libsvn_wc/pristine-store-test.c,

subversion/tests/libsvn_wc/wc-queries-test.c,

subversion/tests/libsvn_wc/wc-test.c:

(max_threads): Renamed from svn_test_max_threads and made static.

(test_funcs): Made static.

(SVN_TEST_MAIN): Expanded boilerplate main() function.

  1. … 54 more files in changeset.
Re-do a portion of r1571499 that broke the build (and was therefore hotfix-

reverted in r1571500).

* subversion/tests/libsvn_subr/named_atomic-test-common.h

(name_namespace): Mark static.

(name_namespace1, name_namespace2): Remove as unused, and..

* subversion/tests/libsvn_subr/named_atomic-test.c

(name_namespace1, name_namespace2): .. declare them here, instead, as static.

  1. … 1 more file in changeset.
Reduce execution time of some C tests by reducing the number of

iterations to a more reasonable level.

* subversion/tests/libsvn_fs_x/string-table-test.c

(many_strings_table_body): Having over 50 sub-tables is no better

than 6 sub-tables when we simply want

to test for large strings and multiple

sub-tables.

* subversion/tests/libsvn_subr/named_atomic-test.c

(calibrate_iterations): Limiting the number of iterations to 250k

still gives plenty of opportunity for faulty

sync to show up.

  1. … 1 more file in changeset.
Add "--parallel" option to our C tests.

Besides shaving off a few seconds off our total test execution time,

parallel execution also stresses our code and the tests themselves

harder. It uncovers things like improperly separated working sets

that make post-failure analysis harder.

Parallel mode is only available with APR 1.3+ and threads enabled.

The option will simply be ignored in other configurations. Further

limitations are cleanups being delayed to after all tests completed

and segfaults not being intercepted.

Some tests can't be executed in parallel. Therefore, we introduce

svn_test_max_threads a simple upper limit to the number of threads

supported that each test must define alongside the test_funcs array.

This patch sets it to "1" in many cases because the tests are so

small that multi-threaded execution is simply not worth it.

* build/run_tests.py

(TestHarness._run_c_test): Pass --parallel option to the individual

test applications.

* subversion/tests/svn_test.h

(svn_test_max_threads): Declare new external setting to be set by

every test application.

* subversion/tests/svn_test_main.c

(parallel): New command line option presence flag.

(parallel_opt,

cl_options): Declare the new "--parallel" option.

(svn_test_add_dir_cleanup): Synchronize shared pool access since

this might be called from multiple

threads at the same time.

(log_results): Result logger function factored out from do_test_num

to be shared between serialized and parallel test

execution code.

(do_test_num): Use that new function.

(test_params_t,

test_thread,

do_tests_concurrently): Threads, their parameter and the thread

starter for thread-pooled test execution.

(main): Handle the new option.

Call concurrent execution when in PARALLEL mode.

* subversion/tests/libsvn_delta/random-test.c

subversion/tests/libsvn_subr/named_atomic-test.c

(svn_test_max_threads): These tests don't support concurrent execution.

* subversion/tests/libsvn_delta/random-test.c

subversion/tests/libsvn_delta/window-test.c

subversion/tests/libsvn_diff/parse-diff-test.c

subversion/tests/libsvn_ra_local/ra-local-test.c

subversion/tests/libsvn_ra/ra-test.c

subversion/tests/libsvn_subr/auth-test.c

subversion/tests/libsvn_subr/cache-test.c

subversion/tests/libsvn_subr/checksum-test.c

subversion/tests/libsvn_subr/compat-test.c

subversion/tests/libsvn_subr/config-test.c

subversion/tests/libsvn_subr/crypto-test.c

subversion/tests/libsvn_subr/dirent_uri-test.c

subversion/tests/libsvn_subr/error-code-test.c

subversion/tests/libsvn_subr/error-test.c

subversion/tests/libsvn_subr/hashdump-test.c

subversion/tests/libsvn_subr/mergeinfo-test.c

subversion/tests/libsvn_subr/named_atomic-test.c

subversion/tests/libsvn_subr/opt-test.c

subversion/tests/libsvn_subr/packed-data-test.c

subversion/tests/libsvn_subr/path-test.c

subversion/tests/libsvn_subr/prefix-string-test.c

subversion/tests/libsvn_subr/priority-queue-test.c

subversion/tests/libsvn_subr/revision-test.c

subversion/tests/libsvn_subr/root-pools-test.c

subversion/tests/libsvn_subr/skel-test.c

subversion/tests/libsvn_subr/spillbuf-test.c

subversion/tests/libsvn_subr/sqlite-test.c

subversion/tests/libsvn_subr/stream-test.c

subversion/tests/libsvn_subr/string-test.c

subversion/tests/libsvn_subr/subst_translate-test.c

subversion/tests/libsvn_subr/time-test.c

subversion/tests/libsvn_subr/utf-test.c

subversion/tests/libsvn_wc/conflict-data-test.c

subversion/tests/libsvn_wc/wc-queries-test.c:

(svn_test_max_threads): These are too small to benefit from multi-threading.

* subversion/tests/libsvn_client/client-test.c

subversion/tests/libsvn_diff/diff-diff3-test.c

subversion/tests/libsvn_fs_base/changes-test.c

subversion/tests/libsvn_fs_base/fs-base-test.c

subversion/tests/libsvn_fs_base/strings-reps-test.c

subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

subversion/tests/libsvn_fs/fs-test.c

subversion/tests/libsvn_fs/locks-test.c

subversion/tests/libsvn_fs_x/fs-x-pack-test.c

subversion/tests/libsvn_fs_x/string-table-test.c

subversion/tests/libsvn_repos/repos-test.c

subversion/tests/libsvn_subr/io-test.c

subversion/tests/libsvn_subr/translate-test.c

subversion/tests/libsvn_wc/db-test.c

subversion/tests/libsvn_wc/op-depth-test.c

subversion/tests/libsvn_wc/wc-test.c

(svn_test_max_threads): Set concurrency limit to whatever various test

runs suggested as the most efficient value.

* subversion/tests/libsvn_wc/entries-compat.c

subversion/tests/libsvn_wc/pristine-store-test.c

(svn_test_max_threads): Set concurrency limit to "unbounded".

  1. … 54 more files in changeset.
Reduce the time spent in named atomic tests. Modern multi-core CPUs still

execute 10s of thousands of iterations even with the reduced timings.

* subversion/tests/libsvn_subr/named_atomic-test.c

(calibrate_iterations): quicker calibration; reduce actual test for .2 secs

(calibrate_concurrency): 8x concurrency is good enough as a limit

Use SVN_VA_NULL instead of plain NULL in variable-arguments lists in all

remaining cases, as far as I can discover, following r1536307 (introducing

SVN_VA_NULL) and r1543394 (warning about missing or inappropriate sentinels for

some Subersion functions).

[Lots of implementation files]: Replace NULL with SVN_VA_NULL in function

calls with variable argument lists.

  1. … 37 more files in changeset.
* **/**

Run tools/dev/remove-trailing-whitespace.sh to remove all trailing whitespace

before we branch 1.8, like we did before creating previous branches.

No functional changes.

  1. … 199 more files in changeset.
Following up on r868009, replace the new invocations of apr_pool_* with

the equivalent svn_pool_* functions for code consistency.

As this is just a macro there are no functional changes.

* subversion/bindings/cxxhl/src/exception.cpp

* subversion/libsvn_client/merge.c

* subversion/libsvn_fs_fs/fs_fs.c

* subversion/libsvn_fs_fs/tree.c

* subversion/libsvn_subr/cache-membuffer.c

* subversion/libsvn_subr/cache_config.c

* subversion/libsvn_subr/mergeinfo.c

* subversion/svnmucc/svnmucc.c

* subversion/tests/libsvn_subr/named_atomic-test.c

* subversion/tests/svn_test_main.c

* tools/dev/fsfs-access-map.c

* tools/server-side/fsfs-reorg.c

* tools/server-side/fsfs-stats.c

(*): Use svn_pool_clear() and svn_pool_destroy() macros.

  1. … 11 more files in changeset.
Fix file access conflict in tests under Windows: cleanup temp files

only after the named atomics have been closed.

* subversion/tests/libsvn_subr/named_atomic-test.c

(cleanup_test_shm): turn into an APR pool cleanup function

(init_test_shm): register the cleanup

(test_basics,

test_bignums,

test_multiple_atomics,

test_namespaces,

test_multithreaded,

test_multiprocess): remove explicit cleanup function calls

Change the way we implement shared memory setup for our named atomics.

Instead of using APR's SHM API, we create a persistent file and simply

mmap it.

The only downside to this is that the SHM file does not get cleaned up

automatically anymore. Therefore, we need to update tests and hotcopy

code.

The underlying issue has been analyzed in this thread:

http://svn.haxx.se/dev/archive-2012-10/0423.shtml

* subversion/include/private/svn_named_atomic.h

(svn_atomic_namespace__cleanup): declare new API

* subversion/libsvn_subr/named_atomic.c

(): update global docstring

(svn_atomic_namespace__create): create a persistent file; mmap it

(svn_atomic_namespace__cleanup): implement new API

* subversion/libsvn_fs_fs/fs_fs.c

(cleanup_revprop_namespace): new utility function

(hotcopy_body): fix comment; remove temp atomics files

* subversion/tests/cmdline/svnadmin_tests.py

(check_hotcopy_fsfs): don't compare temp files related to atomics

* subversion/tests/libsvn_subr/named_atomic-test.c

(cleanup_test_shm): new cleanup function

test_basics,

test_bignums,

test_multiple_atomics,

test_namespaces,

test_multithreaded,

test_multiprocess): call the new function

  1. … 4 more files in changeset.
Add a blank line between declarations and statements where r1352068 didn't.

* subversion/libsvn_client/deprecated.c,

subversion/libsvn_client/info.c,

subversion/libsvn_client/merge.c,

subversion/libsvn_fs_fs/fs_fs.c,

subversion/libsvn_wc/adm_ops.c,

subversion/libsvn_wc/props.c,

subversion/libsvn_wc/upgrade.c,

subversion/libsvn_wc/wc_db.c,

subversion/libsvn_wc/wc_db_util.c,

subversion/tests/libsvn_subr/named_atomic-test.c,

subversion/tests/libsvn_wc/op-depth-test.c:

Add blank lines.

  1. … 10 more files in changeset.
Do not use non-constant initializers in struct variables, since this

violates C'89 and C'90. Although most compilers used for Subversion support

non-constant initializers, some do not, such as Solaris SunPRO.

Patch by: Daniel Richard G. <skunk{_AT_}iSKUNK.ORG>

(tweaked by me: added a similar change in auth.c)

* subversion/libsvn_client/deprecated.c

(svn_client_propset3, svn_client_status4): Initialize struct variable

fields separately when they are non-constant.

* subversion/libsvn_client/info.c

(svn_client_info3): Same.

* subversion/libsvn_client/merge.c

(merge_locked, do_symmetric_merge_locked): Same.

* subversion/libsvn_diff/diff_file.c

(svn_diff_file_options_parse): Same.

* subversion/libsvn_fs_fs/fs_fs.c

(set_cached_window, svn_fs_fs__set_entry): Same.

* subversion/libsvn_fs_fs/lock.c

(walk_locks): Same.

* subversion/libsvn_subr/auth.c

(svn_auth_get_platform_specific_provider): Same.

* subversion/libsvn_subr/sqlite.c

(wrapped_func): Same.

* subversion/libsvn_wc/adm_ops.c

(svn_wc_get_changelists): Same.

* subversion/libsvn_wc/props.c

(svn_wc__prop_list_recursive, svn_wc_prop_set4): Same.

* subversion/libsvn_wc/upgrade.c

(svn_wc__upgrade_sdb): Same.

* subversion/libsvn_wc/wc_db.c

(svn_wc__db_op_set_changelist, svn_wc__db_revert_list_read, read_url,

svn_wc__db_revision_status): Same.

* subversion/libsvn_wc/wc_db_util.c

(svn_wc__db_with_txn): Same.

* subversion/svn/merge-cmd.c

(merge_reintegrate): Same.

* subversion/tests/libsvn_subr/named_atomic-test.c

(proc_found, run_procs): Same.

* subversion/tests/libsvn_wc/op-depth-test.c

(wc_update, check_db_rows, copy_subtest_t, wc_wc_copies, repo_wc_copies):

Same.

  1. … 15 more files in changeset.
To the named atomics API, add a method that checks whether the full

API functionality will be available to the current process.

* subversion/include/private/svn_named_atomic.h

(svn_named_atomic__is_supported): declare new API function

* subversion/libsvn_subr/named_atomic.c

(svn_named_atomic__is_supported): implement new API function

* subversion/tests/libsvn_subr/named_atomic-test.c

(has_sufficient_privileges): drop

(init_test_shm): use new API

Suggested by: gstein

  1. … 2 more files in changeset.
Fix race condition in concurrency tests with larger numbers of parallel procs.

For small iteration counts, the first few procs may run and exit before the

remaining ones even get started. Thus, they might not share the same SHM

(gets deleted and created anew) and won't see the tokens and counter values

created by the first set of procs.

To avoid that situation, the calling test function will keep an instance of the

namespace alive for as long as the test runs.

* subversion/tests/libsvn_subr/named_atomic-test-common.h

(test_pipeline): require the atomics, i.e the namespace, to previously exist

* subversion/tests/libsvn_subr/named_atomic-test.c

(init_concurrency_test_shm): create namespace in calling pool

(calibrate_iterations): drop namespace after each iteration

  1. … 1 more file in changeset.
* subversion/tests/libsvn_subr/named_atomic-test.c

(init_test_shm): ensure that the temp files get cleaned up afterwards

Found by: danielsh

Avoid skewing the test numbers when !APR_HAS_THREADS.

* subversion/tests/libsvn_subr/named_atomic-test.c

(test_multithreaded): Always define, but skip when previously undefined.

(test_funcs): Always call test_multiprocess().

Trim trailing whitespace.

* subversion/include/private/svn_named_atomic.h,

* subversion/include/svn_pools.h,

* subversion/libsvn_client/commit.c,

* subversion/libsvn_client/diff.c,

* subversion/libsvn_client/merge.c,

* subversion/libsvn_fs_fs/dag.c,

* subversion/libsvn_fs_fs/fs.h,

* subversion/libsvn_fs_fs/fs_fs.c,

* subversion/libsvn_fs_fs/temp_serializer.c,

* subversion/libsvn_fs_fs/tree.c,

* subversion/libsvn_repos/rev_hunt.c,

* subversion/libsvn_subr/cache-membuffer.c,

* subversion/libsvn_subr/checksum.c,

* subversion/libsvn_subr/crypto.c,

* subversion/libsvn_subr/named_atomic.c,

* subversion/libsvn_subr/pool.c,

* subversion/libsvn_wc/wc_db.c,

* subversion/svnserve/serve.c,

* subversion/tests/libsvn_subr/named_atomic-test-common.h,

* subversion/tests/libsvn_subr/named_atomic-test-proc.c,

* subversion/tests/libsvn_subr/named_atomic-test.c,

* subversion/tests/libsvn_subr/string-test.c:

Trim trailing whitespace.

  1. … 21 more files in changeset.
* subversion/tests/libsvn_subr/named_atomic-test.c

(init_test_shm): use svn_io_open_unique_file3 to generate temp names

Suggested by: rhuijben

Some build bots have *very* inconsistent performance. They go as low

as 27 iterations per 5 sec (1000+ are normal). Therefore, relax timing

constraints to make those bots unlikely to time out.

* subversion/tests/libsvn_subr/named_atomic-test-common.h

(test_pipeline): raise dealine from 5s to 20s

* subversion/tests/libsvn_subr/named_atomic-test.c

(calibrate_iterations): start at 10 iterations instead of 100

(calibrate_concurrency): default to 100 iterations instead of 200

  1. … 1 more file in changeset.
* subversion/tests/libsvn_subr/named_atomic-test.c

(adjust_proc_path): directory return value must have a longer lifetime

than the function call

* subversion/tests/libsvn_subr/named_atomic-test.c

(run_procs): print detailed parameter list when the sub-process failed to start

Improve error reporting in named atomic tests and lower the calibration

threshold as some build bots fail to reach 200 iterations/s.

* subversion/tests/libsvn_subr/named_atomic-test-common.h

(check_watchdog): if deadline is reached, log progress as part of the error

* subversion/tests/libsvn_subr/named_atomic-test.c

(run_procs): aggregate errors over all sub-processes

(calibrate_iterations): start with 100 iterations since some build bots

seem to take more than 5s with an initial value of 1000.

  1. … 1 more file in changeset.
* subversion/tests/libsvn_subr/named_atomic-test.c

(adjust_proc_path): fix Windows build

Skip test execution in named atomic test when preconditions are not satisfied.

* subversion/tests/libsvn_subr/named_atomic-test.c

(adjust_proc_path): new utility extracted from run_procs

(proc_found): new check function

(init_test_shm): skip test when SHM functionality is not accessible

(run_procs): adapt, fix comments

(calibrate_concurrency): run procs only when available

(test_basics, test_bignums, test_multiple_atomics, test_namespaces,

test_multithreaded): remove local checks

(test_multiprocess): skip if proc is not available

* subversion/tests/libsvn_subr/named_atomic-test-proc.c

(main): make execution with no parameters a no-op

  1. … 1 more file in changeset.
Fix locking scheme for named atomics. We must must ensure that all instances

of the same namespace use the same thread-synchronization object where

thread sync. is required. This will now be achieved by a process-global mutex.

* subversion/libsvn_subr/svn_named_atomic.c

(): check for WIN32 instead of _WIN32 for consistency with the remaining code

(mutex_t): remove namespace-local thread mutex

(USE_THREAD_MUTEX): new constant

(thread_mutex): new static thread mutex

(init_thread_mutex): its init function

(lock, unlock, svn_atomic_namespace__create): adapt

* subversion/tests/libsvn_subr/named_atomic-test.c

(test_funcs): un-WIMP test_multithreaded

  1. … 1 more file in changeset.
Consolidate error messages from named atomics test #6 into

the main error log.

* subversion/tests/libsvn_subr/named_atomic-test.c

(run_procs): re-direct sub-process' stdout to the parent's stdout

* subversion/tests/libsvn_subr/named_atomic-test-proc.c

(main): log errors into stdout instead of stderr, log process id

  1. … 1 more file in changeset.
* subversion/tests/libsvn_subr/named_atomic-test.c

(test_funcs): Mark test_multithreaded WIMP on Unix.

* subversion/tests/libsvn_subr/named_atomic-test.c

fix usage of APR_HAS_THREADS

* subversion/tests/libsvn_subr/named_atomic-test.c

(test_bignums, test_multiple_atomics, test_namespaces,

test_multithreaded, test_multiprocess): Add initialisation so tests

can be run individually.