Checkout Tools
  • last updated 54 mins ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Merge the 1.10-cache-improvements development branch to trunk and remove

the BRANCH-README file. There were no conflicts.

Changes brought by this branch:

- Introduce a NULL cache implementation to unify / simplify cache usage

- Space and runtime efficiency improvements to the recently introduced

full key comparison in membuffer caches.

  1. … 9 more files in changeset.
Add a function to clear a membuffer cache instance.

That is very useful if you want to replace repositories on disk and must

make sure that there is no residual data from the old repository contents

in your caches.

* subversion/include/private/svn_cache.h

(svn_cache__membuffer_clear): Declare new private API function.

* subversion/libsvn_subr/cache-membuffer.c

(svn_cache__membuffer_clear): Implement it.

* subversion/tests/libsvn_subr/cache-test.c

(test_membuffer_cache_clearing): New test.

(test_funcs): Register new test.

  1. … 2 more files in changeset.
Switch cache creation APIs to the two-pool paradigm if they use temporaries.

Update callers, FSFS cache init code in particular.

* subversion/include/private/svn_cache.h

(svn_cache__make_memcache_from_config,

svn_cache__create_membuffer_cache): Add a scratch pool parameter.

* subversion/libsvn_subr/cache-memcache.c

(svn_cache__make_memcache_from_config): Replace the temporary sub-pool

with the new scratch pool.

* subversion/libsvn_subr/cache-membuffer.c

(svn_cache__create_membuffer_cache): Allocate temporaries in the scratch

pool.

* subversion/libsvn_fs_fs/caching.c

(create_cache): Switch to two-pool paradigm and pass them along

to the modified APIs.

(svn_fs_fs__initialize_caches): Provide the scratch pool.

(svn_fs_fs__initialize_txn_caches): Trivially adapt to API change.

* subversion/libsvn_fs_fs/fs_fs.c

(read_config): Provide a scratch pool.

* subversion/libsvn_fs_x/caching.c

(read_config): Same.

(create_cache): Trivially adapt to API change.

* subversion/tests/libsvn_subr/cache-test.c

(test_memcache_basic,

test_membuffer_cache_basic,

test_memcache_long_key): Same.

* tools/server-side/fsfs-stats.c

(read_revisions): Same.

  1. … 7 more files in changeset.
Update documentation of cache value types, following the memcache fixes

in r1570642 and r1585686 which involved a change from svn_string_t to

svn_stringbuf_t.

* subversion/include/private/svn_cache.h

(svn_cache__create_inprocess): Add missing documentation.

(svn_cache__create_memcache,

svn_cache__create_membuffer_cache): Update doc strings.

* subversion/libsvn_fs_fs/fs.h

(fs_fs_data_t): Update the documentation of 'fulltext_cache'.

* subversion/libsvn_fs_x/fs.h

(fs_x_data_t): Same.

  1. … 2 more files in changeset.
* subversion/include/private/svn_cache.h

(SVN_CACHE__MEMBUFFER_DEFAULT_PRIORITY,

SVN_CACHE__MEMBUFFER_LOW_PRIORITY): Fix spelling in docstrings.

No functional change.

Merge revisions r1442051,1442504,1443803,1444690,1477166,1478055,1490674

from branches/fsfs-format7. These introduce the notion of priorities to our

caches and plus comprise all other caching improvements.

A few conflicts had to be resolved.

  1. … 9 more files in changeset.
Improve cache statistics in particular for membuffer caches:

- show index bucket usage distribution

- show size stats only as an option

- provide global membuffer stats

Update our reporting code to use all these nice features.

* subversion/include/private/svn_cache.h

(svn_cache__info_t): add bucket fill level histogram

(svn_cache__get_info): add option to reduce output

(svn_cache__membuffer_get_global_info): new private API function

* subversion/libsvn_subr/cache.c

(svn_cache__format_info): extend implementation

* subversion/libsvn_fs_fs/caching.c

(dump_cache_statistics): access stats only; skip unused caches

(dump_global_cache_statistics): new utility function

(svn_fs_fs__initialize_caches): register global stats dump

* subversion/libsvn_subr/cache-membuffer.c

(svn_membuffer_get_segment_info): optionally include histogram

(svn_membuffer_cache_get_info): update caller

(svn_membuffer_get_global_segment_info): new utility

(svn_cache__membuffer_get_global_info): implement new private API

  1. … 3 more files in changeset.
* **/**

Run tools/dev/remove-trailing-whitespace.sh to remove all trailing whitespace

before we branch 1.8, like we did before creating previous branches.

No functional changes.

  1. … 199 more files in changeset.
Merge forth batch of changes from the 10Gb branch.

Faster membuffer cache lookup and concurrency improvement.

Revisions 1388816,1389044,1389289,1390216

  1. … 9 more files in changeset.
Private API change in svn_cache__t-related serialization functions:

Refer to memory chunks as "void*" instead of "char*". This is more

in line with clib functions like memcpy or malloc and should therefore

be less confusing to readers.

Note: Most of these changes affect function signatures only, i.e.

the function body remains untouched.

* subversion/include/private/svn_cache.h

(svn_cache__deserialize_func_t, svn_cache__partial_getter_func_t,

svn_cache__partial_setter_func_t, svn_cache__serialize_func_t):

replace char* with void* in function type declarations

* subversion/libsvn_fs_fs/dag.h

(svn_fs_fs__dag_serialize, svn_fs_fs__dag_deserialize):

adapt signatures of (de-)serialization functions

* subversion/libsvn_fs_fs/dag.c

(svn_fs_fs__dag_serialize, svn_fs_fs__dag_deserialize): dito

* subversion/libsvn_fs_fs/temp_serializer.h

(svn_fs_fs__serialize_txdelta_window, svn_fs_fs__deserialize_txdelta_window,

svn_fs_fs__serialize_manifest, svn_fs_fs__deserialize_manifest,

svn_fs_fs__serialize_properties, svn_fs_fs__deserialize_properties,

svn_fs_fs__serialize_id, svn_fs_fs__deserialize_id,

svn_fs_fs__serialize_node_revision, svn_fs_fs__deserialize_node_revision,

svn_fs_fs__serialize_dir_entries, svn_fs_fs__deserialize_dir_entries,

svn_fs_fs__get_sharded_offset, svn_fs_fs__extract_dir_entry,

svn_fs_fs__replace_dir_entry): dito

* subversion/libsvn_fs_fs/temp_serializer.c

(svn_fs_fs__serialize_txdelta_window, svn_fs_fs__deserialize_txdelta_window,

svn_fs_fs__serialize_manifest, svn_fs_fs__deserialize_manifest,

svn_fs_fs__serialize_properties, svn_fs_fs__deserialize_properties,

svn_fs_fs__serialize_id, svn_fs_fs__deserialize_id,

svn_fs_fs__serialize_node_revision, svn_fs_fs__deserialize_node_revision,

return_serialized_dir_context, svn_fs_fs__serialize_dir_entries,

svn_fs_fs__deserialize_dir_entries, svn_fs_fs__get_sharded_offset,

svn_fs_fs__extract_dir_entry, slowly_replace_dir_entry,

svn_fs_fs__replace_dir_entry): dito

* subversion/libsvn_subr/cache-inprocess.c

(inprocess_cache_set_internal, inprocess_cache_set_partial_internal):

remove char** cast when calling serializers

* subversion/libsvn_subr/cache-membuffer.c

(membuffer_cache_set): temp buffer is now void* instead of char*

(membuffer_cache_set_partial_internal): add explicit cast to void**

before calling the serializer

(serialize_svn_stringbuf, deserialize_svn_stringbuf):

adapt signatures of (de-)serialization functions

* subversion/libsvn_subr/cache-memcache.c

(memcache_set, memcache_set_partial): adapt implementation

  1. … 7 more files in changeset.
Purge trailing whitespace, in the spirit of r1138109.

  1. … 97 more files in changeset.
Extend svn_cache__t API to support NULL keys. Attempts to

store a value under a NULL key will be ignored and get()

requests for NULL keys will always return "not found".

Also, fix a number of API documentation issues in the

affected functions.

* subversion/include/private/svn_cache.h

(svn_cache__get): mention support for NULL keys;

address ### review comment

(svn_cache__set, svn_cache__set_partial):

mention support for NULL keys

(svn_cache__get_partial): ditto; document pool usage

* subversion/libsvn_subr/cache-memcache.c

(memcache_internal_get): return "not found" for NULL keys

(memcache_set): be a no-op for NULL keys

* subversion/libsvn_subr/cache-membuffer.c

(get_group_index): make NULL keys refer to invalid groups

(membuffer_cache_get_partial, membuffer_cache_set_partial):

add check for invalid groups

(combine_key): a NULL user key results in a NULL internal key

* subversion/libsvn_subr/cache-inprocess.c

(inprocess_cache_get_internal, inprocess_cache_get_partial):

return "not found" for NULL keys

(inprocess_cache_set, inprocess_cache_set_partial):

be a no-op for NULL keys

  1. … 3 more files in changeset.
Offer optional thread-safety for membuffer cache frontends

(actual data container / backend is already thread-safe).

The membuffer cache is now on par with inprocess cache

features.

* subversion/include/private/svn_cache.h

(svn_cache__create_membuffer_cache): add thread_safe parameter

* subversion/libsvn_subr/cache-membuffer.c

(svn_membuffer_cache_t): add optional mutex

(svn_membuffer_cache_get, svn_membuffer_cache_set,

svn_membuffer_cache_get_partial, svn_membuffer_cache_set_partial,

svn_membuffer_cache_is_cachable, svn_membuffer_cache_get_info,

membuffer_cache_vtable): mark as non-sync'ing in docstring

(svn_membuffer_cache_get_synced, svn_membuffer_cache_set_synced,

svn_membuffer_cache_get_partial_synced,

svn_membuffer_cache_set_partial_synced): new sync'ing wrappers

around the non-sync'ing implementations

(membuffer_cache_synced_vtable): new vtable for sync'ing caches

(svn_cache__create_membuffer_cache): depending on thread_safe

parameter, initialize mutex and select proper vtable

* subversion/libsvn_fs_fs/caching.c

(create_cache): adapt caller to provided additional parameter

* subversion/tests/libsvn_subr/cache-test.c

(test_membuffer_cache_basic): dito

  1. … 3 more files in changeset.
The semi-annual trailing whitespace cleansing ritual.

* everywhere:

for extsn in c h cpp java py pl rb; do

sed -i -e 's/[ \t]*$//' `find . -name "*.$extsn" | xargs grep '[ \t]$' -l`

done

  1. … 144 more files in changeset.
Typo fix.

* subversion/include/private/svn_cache.h

(svn_cache__get_partial): Similar to svn_cache__get, not svn_cache__set.

Fix Doxygen mark-up related to the use of '@a' to refer to an argument.

  1. … 12 more files in changeset.
Doxygen mark-up tweaks: instead of @ref, use # to refer to a structure or

trailing () to refer to a function.

  1. … 7 more files in changeset.
Clarify pool usage by properly naming the pool parameters in the

svn_cache APIs. This is basically the patch posted here with a few

tweaks on my part: http://svn.haxx.se/dev/archive-2011-05/0578.shtml

* subversion/include/private/svn_cache.h

(svn_cache__deserialize_func_t, svn_cache__partial_getter_func_t,

svn_cache__partial_setter_func_t, svn_cache__serialize_func_t,

svn_cache__create_memcache, svn_cache__make_memcache_from_config,

svn_cache__membuffer_cache_create, svn_cache__create_membuffer_cache,

svn_cache__get, svn_cache__get_partial, svn_cache__get_info,

svn_cache__format_info): the pools in these functions are result pools

(svn_cache__error_handler_t, svn_cache__set_error_handler,

svn_cache__set, svn_cache__iter): the pools in these functions are scratch pools

Patch by: danielsh

Make the declaration of svn_cache__deserialize_func_t match its

actual usage: the DATA buffer going in is already a local copy and

the function should modify that buffer directly.

* include/private/svn_cache.h

(svn_cache__deserialize_func_t): remove "const"; adapt docstring

* subversion/libsvn_fs_fs/temp_serializer.h

(svn_fs_fs__deserialize_txdelta_window, svn_fs_fs__deserialize_manifest,

svn_fs_fs__deserialize_id, svn_fs_fs__deserialize_node_revision,

svn_fs_fs__deserialize_dir_entries): follow the prototype declaration

* subversion/libsvn_fs_fs/temp_serializer.c

(svn_fs_fs__deserialize_txdelta_window, svn_fs_fs__deserialize_manifest,

svn_fs_fs__deserialize_id, svn_fs_fs__deserialize_node_revision,

svn_fs_fs__deserialize_dir_entries): dito

* subversion/libsvn_fs_fs/dag.h

(svn_fs_fs__dag_deserialize): dito

* subversion/libsvn_fs_fs/dag.c

(svn_fs_fs__dag_deserialize): dito

* subversion/tests/libsvn_subr/cache-test.c

(deserialize_revnum): dito

Suggested by: danielsh

  1. … 5 more files in changeset.
* subversion/include/private/svn_cache.h

(svn_cache__set_error_handler): Clarify the docstring.

Moving membuffer cache related configuration code from deep inside the FS

layer to the svn utilities where all other cache related code lives.

Details on the discussion can be found here:

http://svn.haxx.se/dev/archive-2011-04/0169.shtml

To minimize the risks of breaking builds on other platforms, library

dependencies will be cleaned up in a follow-up commit.

* subversion/include/private/svn_fs_private.h

(svn_fs__get_global_membuffer_cache): remove here

* subversion/include/private/svn_cache.h

(svn_cache__get_global_membuffer_cache): rename and move it to here

* subversion/include/svn_fs.h

(svn_fs_cache_config_t, svn_fs_get_cache_config, svn_fs_set_cache_config):

remove here

* subversion/include/svn_cache_config.h

(svn_cache_config_t, svn_get_cache_config, svn_set_cache_config):

rename and move them to this new header file; adapt docstrings

* subversion/libsvn_fs_util/caching.c

move to:

* subversion/libsvn_subr/svn_cache_config.c

(svn_cache_config_t, svn_get_cache_config, svn_set_cache_config):

rename

* subversion/libsvn_fs_fs/caching.c

(svn_fs_fs__initialize_caches, svn_fs_fs__initialize_txn_caches):

adapt to renamed API functions

* subversion/libsvn_ra/ra_loader.c

(svn_ra_initialize): remove cache config code

* subversion/svn/main.c

(main): move it here and adapt it to renamed API

* subversion/svnadmin/main.c

(#includes): include svn_cache_config.h instead of svn_fs.h

(main): adapt to renamed API functions and structures

* subversion/svnserve/main.c

(#includes, main): dito

* build.conf

(libsvn_subr): add the new svn_cache_config.h header to the DLL exports

  1. … 11 more files in changeset.
Fix conversion and potential overflow warnings in cache stats:

caches may be outside the current process -> use 64 bits for sizes

and counts as well as doubles for internal calculations.

* subversion/include/private/svn_cache.h

(svn_cache__info_t): extend all sizes to 64 bits

* subversion/libsvn_subr/cache.c:

(svn_cache__format_info): adjust format strings,

use doubles for calculations

  1. … 1 more file in changeset.
In symmetry to svn_cache__get_partial, introduce a partial

setter / modifier API function that enabled the caller to

modify changed items without the full deserialization -

modification - serialization chain. Therefore, adding entries

to e.g. directories becomes O(1) instead of O(n).

* subversion/include/private/svn_cache.h

(svn_cache__partial_setter_func_t): declare new callback

(svn_cache__set_partial): declare new API function

* subversion/libsvn_subr/cache.h

(svn_cache__vtable_t): add entry for new API function

* subversion/libsvn_subr/cache.c

(svn_cache__set_partial): implement

* subversion/libsvn_subr/cache-inprocess.c

(inprocess_cache_set_partial): implement for inprocess cache

(inprocess_cache_vtable): extend vtable

* subversion/libsvn_subr/cache-memcache.c

(memcache_set_partial): implement for memcache cache

(memcache_vtable): extend vtable

* subversion/libsvn_subr/cache-membuffer.c

(svn_membuffer_cache_set_partial, membuffer_cache_set_partial):

implement for membuffer cache

(membuffer_cache_vtable): extend vtable

* subversion/libsvn_fs_fs/caching.c

[Committed by accident. Reverted in r1093992.]

  1. … 6 more files in changeset.
Clarify the content of the 'found' return parameter. The problem is

that partial getters perform two lookups: the whole entry by key

and then the sub-item defined by baton and the deserializer logic.

* subversion/include/private/svn_cache.h

(svn_cache__get_partial): in docstring, clarify the meaning of 'found'

* subversion/libsvn_subr/cache-membuffer.c

(membuffer_cache_get_partial): make usage of 'found' comply to

API definition

  1. … 1 more file in changeset.
Adapt docstring to latest function signature changes.

* subversion/include/private/svn_cache.h

(svn_cache__create_inprocess): document id parameter,

reflect replacement of dup_func

Found by: danielsh

Add internal statistics functions to SVN cache API.

* subversion/include/private/svn_cache.h

(svn_cache__info_t): new structure

(svn_cache__get_info, svn_cache__format_info): new internal API

* subversion/libsvn_subr/cache.h

(svn_cache__vtable_t): extend vtable to cover the new function

* subversion/libsvn_subr/cache.c

(svn_cache__get_info, svn_cache__format_info): implement new API

* subversion/libsvn_subr/cache-inprocess.c

(inprocess_cache_get_info): implement new API

(inprocess_cache_vtable): extend vtable

* subversion/libsvn_subr/cache-membuffer.c

(svn_membuffer_cache_t, svn_cache__create_membuffer_cache):

store id as well

(svn_membuffer_cache_get_info): implement new API

(membuffer_cache_vtable): extend vtable

* subversion/libsvn_subr/cache-memcache.c

(memcache_get_info): implement new API

(memcache_vtable): extend vtable

  1. … 5 more files in changeset.
Prepare for the introduction of a generic cache statistics API:

make cache-inprocess caches identifiable.

* subversion/include/private/svn_cache.h

(svn_cache__create_inprocess): add id parameter

* subversion/libsvn_subr/cache-inprocess.c

(inprocess_cache_t): add id member

(svn_cache__create_inprocess): store id parameter

* subversion/libsvn_fs_fs/caching.c

(init_callbacks): new function; factored out from the init function

(svn_fs_fs__initialize_caches): provide IDs / prefixes to all cache types,

call new init sub-function

* subversion/libsvn_fs_fs/tree.c

(make_txn_root): adapt to internal API change

* subversion/tests/libsvn_subr/cache-test.c

(test_inprocess_cache_basic): dito

  1. … 4 more files in changeset.
Header cleanup.

* subversion/include/private/svn_cache.h

(svn_cache__dup_func_t): drop unused typedef

(svn_cache__serialize_func_t): fix docstring formatting

Add generic statistics collection to the caching code.

* subversion/include/private/svn_cache.h

(svn_cache__get, svn_cache__iter, svn_cache__get_partial):

even getters may change the state of the cache

(e.g. collect access statistics)

* subversion/libsvn_subr/cache.h

(svn_cache__t): add access statistics counters

* subversion/libsvn_subr/cache.c

(handle_error): count errors

(svn_cache__iter): adapt to changed declaration

(svn_cache__get, svn_cache__set, svn_cache__get_partial):

count reads, writes and hits

  1. … 2 more files in changeset.
Merge all changes (r1072558 - r1076552) from the

integrate-partial-getter branch and re-enable

svn_membuffer_cache_get_partial().

These patches introduce read access to just parts

of a cached item instead of the whole thing. Use

that to make cached dirent access O(1).

  1. … 12 more files in changeset.