transform_sql.py

Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Remove one source of unnecessary variation between tarballs for

different Subversion releases.

* build/transform_sql.py

(main): Avoid putting the full path to token-map.h in the "automatically

generated" comment as this varies depending on the local path. The

other filename in the comment is already simply a basename.

Require Python ≥2.7 for Subversion 1.9, part 2: coding simplifications.

* build/generator/gen_win.py (hashlib_md5),

* build/transform_sql.py (operator.methodcaller),

* subversion/bindings/swig/python/svn/core.py (SubversionException.message),

* tools/dev/scramble-tree.py (hashlib_md5)

* tools/dist/release.py (__future__.with_statement):

Remove conditional code for now-unsupported Python versions.

* subversion/bindings/ctypes-python/csvn/ext/callback_receiver.py

Note a possible code simplification.

  1. … 5 more files in changeset.
autogen.sh: Support Python 3. All Python versions previously supported

remain supported.

Review by: rhuijben, brane

* build/generator/gen_base.py

(GeneratorBase.write_file_if_changed):

Convert str to bytes. Document argument type.

* build/getversion.py

(traceback): Import.

(svn_extractor, sqlite_extractor):

Rewrite 'except' statements in a syntax compatible with both Python 2.5

and Python 3. No functional change.

* gen-make.py:

Ditto.

* build/transform_sql.py

(Processor.process_file): Use dict.items() instead of dict.iteritems().

* subversion/tests/libsvn_subr/subst_translate-test.c

(test_svn_subst_translate_string2_null_encoding_helper):

Convert comment to ASCII.

  1. … 4 more files in changeset.
Fix svn_wc__db_op_make_copy_internal()'s handling of mixed revision BASE trees.

This fixes some issues around tree conflict handling, and improves

database consistency.

* build/transform_sql.py

(process_file): Allow passing columns and binding variables as argument of

IS_STRICT_DESCENDANT_OF, like the other macros.

* subversion/libsvn_wc/wc-checks.sql

(STMT_STATIC_VERIFY): Add documentation to the older statements. Extend.

* subversion/libsvn_wc/wc-queries.sql

(STMT_DELETE_WORKING_BASE_DELETE): Split into non-recursive...

(STMT_DELETE_WORKING_BASE_DELETE_RECURSIVE): ... and recursive variants.

(STMT_INSERT_WORKING_NODE_FROM_BASE_COPY): Allow replacing nodes, and

handle moved_to when doing that.

* subversion/libsvn_wc/wc_db.c

(db_base_remove): Update caller.

(make_copy_txn): Add recursion arguments to determine when a new op-depth

is needed. Simplify code, by moving the initial shadowing to the calling

function.

(make_copy_move_moved_to): New function.

(svn_wc__db_op_make_copy_internal): Update caller. Shadow all nodes with

base-deleted before calling make_copy_txn. Move moved-to information

to this layer.

* subversion/tests/cmdline/tree_conflict_tests.py

(update_delete_mixed_rev): Remove XFail. Tweak status value.

* subversion/tests/libsvn_wc/op-depth-test.c

(make_copy_mixed): New function. Tests svn_wc__db_op_make_copy_internal.

make_copy_and_delete_mixed): New functions. Tests svn_wc__db_base_remove

and shows an existing issue with move handling on base-delete.

(test_list): Add new items.

  1. … 5 more files in changeset.
* build/transform_sql.py

(RELPATH_SKIP_JOIN): Extend documentation.

* build/transform_sql.py

(main): Mention token-map.h in the generated file's header.

Make the switch detection code much easier to understand by introducing a few

more preprocessor sql functions.

This is partly a showcase of how we can reduce some c code complexity using

these functions.

* build/transform_sql.py

(process_file): Add RELPATH_JOIN(x, y), RELPATH_SKIP_ANCESTOR(x, y) and

RELPATH_SKIP_JOIN(x, y, z) which is a combination of the

other two.

* subversion/libsvn_wc/wc-queries.sql

(STMT_HAS_SWITCHED): Simplify query. Use additional argument instead of

join.

(STMT_HAS_SWITCHED_REPOS_ROOT,

STMT_HAS_SWITCHED_WCROOT,

STMT_HAS_SWITCHED_WCROOT_REPOS_ROOT): Remove queries that can now be

handled by STMT_HAS_SWITCHED.

* subversion/libsvn_wc/wc_db.c

(has_switched_subtrees): Use single query. Pass repository relative path.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements): Remove specialized selects on the entire wc.

  1. … 3 more files in changeset.
Micro improvement to the autogen-time build system.

* build/transform_sql.py

(extract_token_map): Account for hypothetical MAP_MAP_FOO macro names.

Followup to r1420100: fix building on centos.

* build/transform_sql.py

(operator.methodcaller): Provide a local implementation if a library

implementation is missing.

Refactor code. No functional change.

* build/transform_sql.py

(operator): Import.

(hotspots): New helper, factored out of extract_token_map().

(extract_token_map): Rewrite in functional style.

* build/transform_sql.py

(NonRewritableDict): Expand docstring.

Improve the new symbolic token maps.

* build/transform_sql.py

(NonRewritableDict): New.

(extract_token_map): Ensure that mappings do not override each other.

Improve the new symbolic token maps.

* build/transform_sql.py

(Processor.process_line): Match tokens as full words.

Use symbolic names for string constants.

Suggested by: philip

* subversion/libsvn_wc/token-map.h

(kind_map): Add an initial symbolic name.

* subversion/libsvn_wc/wc-queries.sql: Use it.

* build/transform_sql.py

(extract_token_map): Parse token-map.h.

(main): Build a token map.

(Processor.__init__): Take a token map.

(Processor.process_file): Use it.

  1. … 2 more files in changeset.
Redefine our Sqlite IS_STRICT_DESCENDANT_OF() macro to allow operating on the

"" relpath. This removes the need of duplicating queries for the root and

subtrees, without losing performance on the subtree case.

The full tree would use an index to select on wc_id anyway, so adding

an extra key doesn't slow it down.

* build/transform_sql.py

(process_file): Redefine IS_STRICT_DESCENDANT_OF().

* subversion/libsvn_wc/wc-queries.sql

(STMT_CLEAR_BASE_NODE_RECURSIVE_DAV_CACHE,

STMT_RECURSIVE_UPDATE_NODE_REPO,

STMT_INSERT_TARGET_DEPTH_INFINITY,

STMT_INSERT_TARGET_WITH_CHANGELIST_DEPTH_INFINITY,

STMT_DELETE_WC_LOCK_ORPHAN_RECURSIVE,

STMT_SELECT_COMMITTABLE_EXTERNALS_BELOW,

STMT_SELECT_EXTERNALS_DEFINED,

STMT_SELECT_EXTERNAL_PROPERTIES,

STMT_SELECT_MIN_MAX_REVISIONS,

STMT_HAS_SPARSE_NODES,

STMT_SUBTREE_HAS_TREE_MODIFICATIONS,

STMT_SUBTREE_HAS_PROP_MODIFICATIONS,

STMT_SELECT_BASE_FILES_RECURSIVE):

Remove now unneeded ?X = '' subexpressions.

* subversion/tests/libsvn_wc/wc-queries-test.c

(slow_statements):

Remove statements that started using indexes by this change.

  1. … 2 more files in changeset.
Make wc-queries-test failures much easier to diagnose by extending the sqlite

transform script a bit. Store names and some optional value during .sql

parsing and make that information available in the test.

* build/transform_sql.py

(Processor): Extend regex to capture flags.

(_sub_statement): Generate extra macro.

(main): Generate a DECLARE_STATEMENT_INFO macro.

* subversion/tests/libsvn_wc/wc-queries-test.c

(wc_query_info): New static variable.

(test_parsable,

test_query_expectations): Extend error messages with statement name.

  1. … 1 more file in changeset.
Make STMT_SELECT_BASE_NODE_LOCK_TOKENS_RECURSIVE only retrieve the lock tokens

below the specified path, instead of also for the path itself.

This allows simplifying the query to use parent_relpath. I think this should

improve the sqlite index usage for this query.

* build/transform_sql.py

(process_file): Update generated query to stop searching for strings between

"/" and "0" without a prefix.

* subversion/libsvn_client/commit_util.c

(harvest_status_callback): Don't search for lock tokens if not deleting a

BASE node.

* subversion/libsvn_wc/wc-queries.sql

(STMT_DELETE_SHADOWED_RECURSIVE): Only select descendants.

(STMT_SELECT_BASE_NODE_LOCK_TOKENS_RECURSIVE): Only select descendants.

* subversion/libsvn_wc/wc_db.c

(svn_wc__db_base_get_lock_tokens_recursive): Update comment.

* subversion/libsvn_wc/wc_db.h

(svn_wc__db_base_get_lock_tokens_recursive): Update comment.

  1. … 4 more files in changeset.
Fix SQL transformation on Windows (and NetWare) builds. The \r\n pattern

messed up the output files.

Reported by: NormW <normw{_AT_}gknw.net>

* build/transform_sql.py:

(Processor.process_file): strip trailing whitespace from the line before

placing it into the output. this will clear off a \r character, and

any potential trailing spaces. (continue to add a space in the format

string so that string-joining does not create something like:

"... FROM tableWHERE ...") we keep leading whitespace for nice

indentation of the queries.

* build/transform_sql.py

(Processor.process_file): Document the magic number^W^Wmagic string '0'.

* build/transform_sql.py

(Processor.process_file): Tweak generated parentheses.

Factor out a repeated code fraction, for maintainability and readability.

* build/transform_sql.py

(Processor.process_file):

Grow support for a IS_STRICT_DESCENDANT_OF() macro. At this time the

macro must be contained within one input line.

* subversion/libsvn_wc/wc-queries.sql

(*): Replace the "Is a descendant" check with the macro.

No functional change --- this only adds/removes some parentheses from the

generated file.

  1. … 1 more file in changeset.
Make the sql transformer a bit more pythonic in the way that it handles the

various directives.

* build/transform_sql.py

(_sub_format, _sub_statement, _sub_include, _sub_define):

New.

(__init__): Initialize the directives dictionary.

(process_file): Use the directives dict to iterate over the various sqlite

directives which we support.

Remove the redundant declaration of the current working copy format. This

introduces a new directive in our sqlite files, which allows us to recycle

the define from wc.h.

* build/transform_sql.py

(process_file): Process the new directive.

* subversion/libsvn_wc/wc-metadata.sql

(STMT_CREATE_SCHEMA): Use the new directive to pull in the version number

from the wc.h header file.

  1. … 1 more file in changeset.
* build/transform_sql.py:

(Processor.re_statement): allow numbers within statement names

Shove the main transformation logic into a class, in order to keep state

across recursions. Add support for an 'include' directive. This will allow

us to keep STMT_* definitions in multiple files, yet generate a single set

of statements for a library.

* build/transform_sql.py:

(class Processor): new class, based on the code from main().

(main): gutted, with code moved to Processor. adjust params.

(): pass a file path rather than a handle to main()

svn_sqlite__open() expects a NULL-terminated array of statement strings,

however we were not actually storing a NULL into the array.

* build/transform_sql.py:

(main): add one more array element containing a NULL

Apply ASF license headers to a number of files included under that license

but lacking said header.

Patch by: Niall Pemberton <niallp{_AT_}apache.org>

  1. … 175 more files in changeset.
Fix a cascade of segfaults introduced in r39345, by actually giving valid

statements to the wc-ng sqlite database open handlers. Also, patch the sql

transformer to help prevent this unfortunate circumstance in the future.

* subversion/libsvn_wc/wc_db.c:

Declare queries from the wc-queries.h file.

* build/transform_sql.py

(main): Only allow a DECLARE_STATEMENTS macro in the output if we had

statements.

  1. … 1 more file in changeset.
* build/transform_sql.py: Add an empty line to make the generated *.h files

a bit more readable.

Revamp some sqlite-related functionality: specifying statements, upgrade

handling, and trimming unused APIs.

Of particular note is the statement handling. The SQL statements are now

moved into the .sql file and preprocessed by the transform_sql.py script.

This allows the same statement indexes to be used by *all* users of the

particular schema. The prior idiom could lead to two files with different

ideas of what statement index 5 represented, and based on who opened the

database. We also get nifty SQL syntax highlighting now.

* subversion/include/private/svn_sqlite.h:

(svn_sqlite__upgrade_func_t): removed. no longer used.

(svn_sqlite__open): remove the upgrade callback. callers should handle

upgrading "manually"

(svn_sqlite__transaction_being, svn_sqlite__transaction_commit,

svn_sqlite__transaction_rollback): removed. unused. callers have

been using the with_transaction API.

* subversion/libsvn_subr/sqlite.c:

(svn_sqlite__transaction_being, svn_sqlite__transaction_commit,

svn_sqlite__transaction_rollback): removed.

(struct upgrade_baton): remove UPGRADE_FUNC and UPGRADE_BATON members

(upgrade_format): no longer call the upgrade callback

(check_format): remove the UPGRADE_FUNC and UPGRADE_BATON params. adjust

the upgrade_format baton.

(svn_sqlite__open): remove the UPGRADE_FUNC and UPGRADE_BATON params.

update call to check_format.

(svn_sqlite__with_transaction): inline the SQL statements rather than

using the transaction wrappers

* subversion/tests/libsvn_wc/db-test.c:

(create_fake_wc): remove the upgrade func/baton from the sqlite open call.

* subversion/libsvn_wc/wc-metadata.sql:

(STMT_*): new statements collected from wc_db.c, entries.c, and

upgrade.c. note that wc_db.c and entries.c each defined a slightly

different STMT_INSERT_BASE_NODE. rename entries' version to

STMT_INSERT_BASE_NODE_FOR_ENTRY.

* subversion/libsvn_wc/wc_db.c:

(enum statement_keys): removed

(statements[]): now declared/defined using the DECLARE_STATEMENTS macro

constructed by transform_sql

(svn_wc__db_upgrade_func): removed. unused.

(open_db): small wrapper around sqlite__open to deal with wc-specific

values. no longer pass upgrade_func to the open.

(parse_local_abspath, create_db): switch to use open_db()

(get_statement_for_path): swich STMT_IDX to a simple integer

(svn_wc__db_temp_get_sdb): no longer take a file-specific set of

statements, since these are now "global". use open_db()

* subversion/libsvn_wc/wc_db.h:

(svn_wc__db_upgrade_func): removed.

(svn_wc__db_temp_get_sdb): no longer take statements

* subversion/libsvn_wc/entries.c:

(enum statement_keys, statements[]): removed.

(read_entries_new, svn_wc__entries_write_new): no longer pass statements

to temp_get_sdb().

(insert_base_node): our STMT_INSERT_BASE_NODE was slightly different

from that of wc_db.c, so it was renamed to

STMT_INSERT_BASE_NODE_FOR_ENTRY. track that change.

* subversion/libsvn-wc/upgrade.c:

(...): add some clarifying comments

(migrate_single_tree_conflict_data): no longer take INSERT_STMT since we

can easily grab it from the database.

(migrate_tree_conflicts): no longer manually define/prepare statements,

but just include them into the "global" set of statements. stop

resetting the SELECT and just process all rows at once; SQLite has

been able to support that for a while. wipe out all conflict data in a

single UPDATE statement, rather than one row at a time.

* subversion/libsvn_fs_fs/rep-cache.c:

(enum statement_keys): removed

(statements[]): switch to using DECLARE_STATEMENTS for this

(svn_fs_fs__open_rep_cache): no longer pass UPGRADE_FUNC/BATON to the

sqlite__open call.

* subversoin/libsvn_fs_fs/rep-cache-db.sql:

(STMT_*): new statements, pulled from rep-cache.c

* build/transform_sql.py:

(main): add detection of STMT_* directives, generating appropriate

output for each one. at the end of the generated header, include a new

DECLARE_STATEMENTS macro to declare/define all the statements

  1. … 10 more files in changeset.