subversion

Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1342524 is being indexed.

Avoid nested full table scans on two tables in a single query by rewriting the

query to a completely different pattern.

Only operate on the relevant subset of the actual table to avoid doing unneeded

work on all other nodes in the working copy.

The IN (select local_relpath FROM .. WHERE .. AND kind = 'qqq') style

queries are not optimized by Sqlite to use a select on local_relpath.

Depending on the number of affected nodes and nodes in the working copy, this

query is now orders of magnitude faster.

* subversion/libsvn_wc/wc-queries.sql

(STMT_UPDATE_ACTUAL_CHANGELISTS): Rewrite query.

* subversion/libsvn_wc/wc_db.c

(set_changelist_txn): Pass the operation root as second argument.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove another slow statement from the list.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove the move queries as these use an index now.

Use the logging module a bit more in the Python tests.

* subversion/tests/cmdline/getopt_tests.py,

subversion/tests/cmdline/lock_tests.py,

subversion/tests/cmdline/autoprop_tests.py:

s/print/logger.warn/

A bit more logging goodness.

* subversion/tests/cmdline/svntest/main.py

(wrap_ex, copy_repos): Use the logger to record error conditions.

Use a #if for a compile-time check, rather than an if().

* subversion/tests/libsvn_wc/wc-queries-test.c

(test_sqlite_version): Use the preprocessor when determining which version

of SQLite we're building with, rather than doing so at runtime.

* build/generator/gen_vcnet_vcproj.py

(write): Use a proper join for paths, instead of a string concat.

Add an index on the moved_to field in the NODES table to speed up move

processing in the wc_db api.

The index will be added to new tables, and once we bump to format 30

(presumably before 1.8.0) we will add it to existing tables.

This resolves 3 slow statement cases in the wc-queries-tests, but I'm not sure

if just committing a change to wc-metadata.sql forces a query rebuild.

* subversion/libsvn_wc/wc-metadata.sql

(STMT_CREATE_NODES): Update file external information.

Add creation of I_NODES_MOVED index.

(STMT_UPGRADE_TO_30): New statement with upgrade statement.

* subversion/tests/libsvn_wc/wc-queries-test.c

(supported_explain_query_plan): Following up on r1342243, don't return FALSE

when the query plan is empty, as that is the case for the dumb test query.

* subversion/tests/cmdline/merge_reintegrate_tests.py

(renamed_branch_reintegrate): New test.

(test_list): Run it.

Patch by: me

Review by: Cristian Amarie <cristian.amarie{_AT_}gmail.com>

(Cristian checked my initial attempt and fixed a revision number error.)

* subversion/tests/libsvn_wc/wc-queries-test.c

(supported_explain_query_plan): New function, extracted from

test_query_expectations.

(test_query_expectations): Extract support detection code.

Don't produce errors for unexpected fast queries by adding the

query plan to the error chain.

[And I replaced some accidentally added tabs with spaces]

* subversion/libsvn_wc/wc_db.c

(is_wclocked): Following up on r1342217, remove the assumption that the

first match will be the only match as this is no longer true.

But it is cheaper to check all matches, then to make Sqlite copy all

data for sorting and then check the result. Especially when you have

not that many locks in your working copy.

Found by the entries_compat tests (but missed by me until after I clicked

commit)

It is cheaper for us to read a few more Sqlite results that might even have

the answer we looking for than for Sqlite to collect all items and provide

them sorted.

Apply this knowledge on the is_wclocked check + query.

* subversion/libsvn_wc/wc-queries.sql

(STMT_SELECT_ANCESTOR_WCLOCKS): Reorder some checks for sanity and remove

the order by, as Sqlite doesn't know that the passed arguments are related.

* subversion/tests/libsvn_wc/wc-queries-test.c

(STMT_SELECT_ANCESTOR_WCLOCKS): And another 'slow' query gone.

Provide the Sqlite query plan as part of the error message, to avoid having

to install all kinds of Sqlite versions when problems are reported.

* subversion/tests/libsvn_wc/wc-queries-test.c

(test_query_expectations): Store the already copied rows in an array and

add them to created warnings.

* subversion/libsvn_wc/wc-queries.sql

(STMT_SELECT_GE_OP_DEPTH_CHILDREN): Avoid creating a temporary table

by using UNION ALL instead of UNION.

(Without ALL duplicates are filtered by using a temp table)

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove slow marker.

Following up on r1341889, fix an ordering problem in the changelist

notifications introduced by adding a primary key on the temporary

table.

We really want to notify the removal from a changelist before the notification

of adding it to the new one.

(Unless we would want to change behavior and only show a single list).

Found by the ruby testsuite.

* subversion/libsvn_wc/wc-queries.sql

(changelist_list): Update primary key, to order them correctly on insert.

(STMT_SELECT_CHANGELIST_LIST): Ask for the records in the right order.

wc-queries-test.c ensures that this doesn't cost anything for us, as

the table is pre-sorted.

Stop generating $(SolutionDir) inside project files where we can just calculate

the relative path. This somehow triggered a bug in more recent VS versions.

* build/generator/gen_vcnet_vcproj.py

(find_rootpath): Remove function (fall back to base class).

(write): Calculate relative path.

* subversion/tests/libsvn_wc/wc-queries-test.c

(test_sqlite_version): Grammar tweak.

Wc-queries-test: Remove dependency on libsvn_wc itself. Add version check

as a separate test, to provide a bit more information.

* build.conf

(wc-queries-test): Remove dependency.

* subversion/tests/libsvn_wc/wc-queries-test.c

(includes): Add private/svn_dep_compat.h.

(test_sqlite_version): New function.

(test_parsable): Remove printf, that is now part of test_sqlite_version.

Extend the wc-queries-test application to catch another type of

potentially slow queries: Queries that create temporary tables.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove STMT_SELECT_CHANGELIST_LIST as that is now

caught as a result table. Add 3 queries that use a temp table.

(explanation_item): Add some variables.

(parse_explanation_item): Parse COMPOUND and some bits of USE. Set

search type.

(is_result_table): Add changelist_list.

(test_query_expectations): Update tests to handle other info items.

Check for search items before testing fields that may be NULL.

* subversion/tests/cmdline/diff_tests.py

(diff_renamed_dir): Extend to cover the changes made in r1341927.

This test will now fail with pre-r1341927 clients.

While here, fix a comment that wrongly described a wc->wc diff

as a repos->wc diff.

Fix repos->wc diff of copied/moved directories, i.e. "svn diff -rN DIR"

where DIR is a locally copied or moved-here directory.

Before this commit, the resulting diff would always show the contents

of DIR as being deleted. Now, a diff against the copy source as of rN

is shown, which matches the behaviour described in 'svn help diff'.

* subversion/libsvn_client/diff.c

(diff_cmd_baton): Add repos_wc_diff_target_is_copy boolean, required

to decide whether revision numbers passed into file_added() and

file_changed() can be trusted.

(diff_file_added, diff_file_changed): During a repos->wc diff of a copied

directory, use revision numbers from the diff baton. Both revisions passed

in are SVN_INVALID_REVNUM because the diff target is a local copy.

(diff_repos_wc): When performing a repos->wc diff of a copied directory,

instead of crawling the working copy to create a report, describe the copy

source to the reporter directly. This way, diffing a copy with local

modifications results in similar output as if the copy source was modified

and then diffed (but the source need not be present in the working copy).

Suggested by: rhuijben

Separate the changelist table create script from the creation of a trigger on

actual_nodes, to avoid side effects when testing in wc-queries-test.c.

As this code is only evaluated within a lock, there is no

measurable performance impact of this split.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CREATE_CHANGELIST_LIST): Separate last part of this schema to...

(STMT_CREATE_CHANGELIST_TRIGGER): ... this. To make sure we don't create

unexpected triggers on common tables in the wc-queries-test.

* subversion/libsvn_wc/wc_db.c

(set_changelist_txn): Add second call to svn_sqlite__exec_statements, to

keep the original behavior.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CREATE_CHANGELIST_LIST): Simplify comment. Inserts aren't used and

the other cases are simple now.

Avoid some unneeded work when setting and clearing changelists.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CREATE_CHANGELIST_LIST): Give table a primary key.

Replace three separate triggers with one simpler one.

* subversion/libsvn_wc/wc_db.c

(set_changelist_txn): Don't create unneeded actual nodes, when we are just

clearing the changelists.

* subversion/tests/cmdline/changelist_tests.py

(verify_changelist_output): First show what changed, before erroring on the

number of lines.

(add_remove_changelists): Use sbox.ospath() to improve readability.

Cleanup the svn ci --include-externals query to use indexes for most of its

work. Use the knowledge that 'parent_relpath = ?' is a subset of

IS_STRICT_DESCENDANT_OF() to at least use some indexing.

* subversion/libsvn_wc/externals.c

(is_external_rolled_out): Externals live in BASE, so use the cheap/direct

api towards that layer, without bothering with copies.

* subversion/libsvn_wc/wc-queries.sql

(STMT_SELECT_COMMITTABLE_EXTERNALS_BELOW): Rewrite query.

Remove unused result column. Add missing wc_id in subquery. Just obtain

the repos_id from the wc root, as we can't trust it to be available on

the passed node.

* subversion/libsvn_wc/wc_db.c

(svn_wc__db_committable_externals_below): Update column indexes.

* subversion/tests/libsvn_wc/wc-queries-test.c

(STMT_SELECT_COMMITTABLE_EXTERNALS_BELOW): And another one is done.

Help the Sqlite query planner a bit by rewriting two queries in a way that

makes it use indexes, where it didn't before.

* subversion/libsvn_wc/wc-queries.sql

(STMT_RECURSIVE_UPDATE_NODE_REPO,

STMT_SELECT_EXTERNALS_DEFINED): Make the OR operation the outer operation

by duplicating some cheap tests.

(STMT_SELECT_COMMITTABLE_EXTERNALS_BELOW): Accidental change, that should

have been part of the r1341851 change.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove two slow statements.

Clean up some loose ends in the streamy property handling.

Rename some statements and apply a generic exception in the statement

validator for tables like the property cache.

No functional changes.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CREATE_NODE_PROPS_CACHE): Rename to...

(STMT_CREATE_TARGET_PROP_CACHE): ... as it is more related to target than

node processing. Remove proof of concept note, as this was part of 1.7.

Rename temporary table to target_prop_cache, to more closely match

our other temporary table names.

Define primary key to avoid a sort when reporting.

(STMT_CACHE_TARGET_PROPS,

STMT_CACHE_TARGET_PRISTINE_PROPS): Update users

(STMT_SELECT_RELEVANT_PROPS_FROM_CACHE): Rename to ...

(STMT_SELECT_ALL_PROP_CACHE):

STMT_DROP_NODE_PROPS_CACHE): Update users.

* subversion/libsvn_wc/wc_db.c

(cache_props_recursive): Update user.

* subversion/tests/libsvn_wc/wc-queries-test.c

(schema_statements): Update reference.

(slow_statements): Remove statement.

(is_node_table): Add final line to allow extending with one line changes.

(is_result_table): New function.

(test_query_expectations): Allow exceptions on scan and index rules on

(temporary) result tables.

Make the recursive property retrieval functions in the wc_db api proportional

to the list of targets instead of to the total size of the working copy.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CREATE_TARGETS_LIST): Add primary key, to give Sqlite a primary index.

(STMT_CACHE_NODE_PROPS): Renamed to ...

(STMT_CACHE_TARGET_PROPS): ... this. Join starting from the targets table

to avoid a wc scan on the nodes table to check for each item if it is

in the targets list. Integrate the STMT_CACHE_ACTUAL_PROPS statement

as that gives us the final result in a single step.

(STMT_CACHE_ACTUAL_PROPS): Remove statement.

Folded into STMT_CACHE_TARGET_PROPS.

(STMT_CACHE_NODE_PRISTINE_PROPS): Renamed to ...

(STMT_CACHE_TARGET_PRISTINE_PROPS): ... this. Join starting with the targets

table. Use a CASE to only fetch properties from below when we have a

base-deleted node.

* subversion/libsvn_wc/wc_db.c

(cache_props_recursive): Remove usage of STMT_CACHE_ACTUAL_PROPS.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove three more slow statements.

Avoid a table scan of ACTUAL_NODES for every directory during wc copy

operations by splitting a query into two separate queries that each

use the right index.

* subversion/libsvn_wc/wc-queries.sql

(STMT_SELECT_CONFLICT_MARKER_FILES): Split this query into ...

(STMT_SELECT_CONFLICT_MARKER_FILES1): ... this, for the directory and ...

(STMT_SELECT_CONFLICT_MARKER_FILES2): ... this, for the nodes within.

* subversion/libsvn_wc/wc_db.c

(get_conflict_marker_files): New function. Using two queries. Extracted

from ...

(svn_wc__db_get_conflict_marker_files): ... this function.

* subversion/libsvn_wc/wc_db.h

(svn_wc__db_get_conflict_marker_files): Remove obsolete comment.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove yet another statement from the slow list.

To remove another slow query, remove a dead code path: streamily retrieving

the op_depth 0 properties.

* subversion/include/private/svn_wc_private.h

(svn_wc__prop_list_recursive): Remove argument.

* subversion/libsvn_client/prop_commands.c

(get_prop_from_wc,

svn_client_proplist3): Update caller.

* subversion/libsvn_wc/props.c

(svn_wc__prop_list_recursive): Remove argument. Update caller.

* subversion/libsvn_wc/wc-queries.sql

(STMT_CACHE_NODE_BASE_PROPS): Remove unused query.

* subversion/libsvn_wc/wc_db.c

(cache_props_baton_t): Remove variable.

(cache_props_recursive): Remove dead code path.

(svn_wc__db_read_props_streamily): Remove argument.

* subversion/libsvn_wc/wc_db.h

(svn_wc__db_read_props_streamily): Remove argument.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove query.