gen_win_dependencies.py

Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1817042 is being indexed.

fsfs: Use the `WITHOUT ROWID` optimization for rep-cache.db in format 8.

This optimization, introduced in SQLite 3.8.2, works well for tables that

have non-integer primary keys, such as

hash TEXT NOT NULL PRIMARY KEY

in the rep-cache.db. (See the https://sqlite.org/withoutrowid.html article

for additional details.)

A quick experiment showed a reduction of the on-disk size of the database

by ~1.75x. The lookups should also be faster, both due to the reduced

database size and due to the lesser amount of internal bsearches. This

should improve the times of new commits and `svnadmin load`, especially

for large repositories that also have large rep-cache.db files.

In order to maintain compatibility, since SQLite versions prior to 3.8.2

do not support this statement, we only start using it for fsfs format 8

repositories and simultaneously bump the minimal required SQLite version

from 3.7.12 (May 2012) to 3.8.2 (December 2013). The last step ensures that

all binaries compiled to support format 8 can work with the tables with

this optimization. Also, as the various scripts have both the minimal

and recommended (3.7.15.1) SQLite versions, we bump the recommended

version to the last 3.8.x patch version, which is 3.8.11.1.

* subversion/libsvn_fs_fs/rep-cache-db.sql

(STMT_CREATE_SCHEMA): Rename this ...

(STMT_CREATE_SCHEMA_V1): ...to this.

(STMT_CREATE_SCHEMA_V2): New, enables `WITHOUT ROWID` optimization.

(STMT_GET_REP, STMT_SET_REP, STMT_GET_REPS_FOR_RANGE,

STMT_GET_MAX_REV, STMT_DEL_REPS_YOUNGER_THAN_REV,

STMT_LOCK_REP, STMT_UNLOCK_REP):

Note that these statements work for both V1 and V2 schemas.

* subversion/libsvn_fs_fs/fs.h

(SVN_FS_FS__MIN_REP_CACHE_SCHEMA_V2_FORMAT): New.

* subversion/libsvn_fs_fs/rep-cache.c

(REP_CACHE_SCHEMA_FORMAT): Remove.

(open_rep_cache): Select between creating a V1 or V2 schemas based

on the format of the filesystem.

* subversion/libsvn_subr/sqlite.c

(): Bump minimum required SQLite version to 3.8.2.

* subversion/tests/cmdline/svnadmin_tests.py

(check_hotcopy_fsfs_fsx): Check if the Python's built-in SQLite version

is enough to interpret the schema of rep-cache.db, and skip the check

if it's not.

* build/generator/gen_win_dependencies.py

(_find_sqlite): Bump minimum required SQLite version to 3.8.2.

* configure.ac

(SQLITE_MINIMUM_VER): Bump to 3.8.2.

(SQLITE_RECOMMENDED_VER): Bump to 3.8.11.1.

(SQLITE_RECOMMENDED_VER_REL_YEAR): New, required to construct the

download URL which includes the release year for the newer SQLite

amalgamation versions.

(SQLITE_URL): Update the download URL.

* get-deps.sh

(SQLITE_VERSION): Bump to 3.8.11.1.

(SQLITE_VERSION_REL_YEAR): New.

(get_sqlite): Update the download URL that includes the release year

for the newer SQLite amalgamation versions.

* INSTALL

(C.12.SQLite): Bump minimum required SQLite version to 3.8.2.

(E.1.Prerequisites): Bump the minimum and recommended SQLite versions.

  1. … 8 more files in changeset.
Following up on r1814724, r1814725, also fix the DLL name for OpenSSL

1.1.0 to allow the test code to automatically copy the required libraries.

* build/generator/gen_win_dependencies.py

(_find_openssl): Generate dll name for 1.1.0+.

* build/generator/gen_win_dependencies.py

Fix typo at start of file introduced in r1814724.

Allow building against OpenSSL 1.1.0 on Windows by properly detect the

library name, that starting with 1.1.0 now matches that on other platforms.

* build/generator/gen_win_dependencies.py

(*): Accidentally added an 'f' at start of file. Fixed in r1814725.

(_find_openssl): Use new library names when possible.

* build.conf

Use standard dependency notation on Windows to support dynamic names.

  1. … 1 more file in changeset.
Fix Windows build with MS Visual Studio 2017.

"It fails to compile because the toolset is v141. It looks for toolset v150

but cannot find them."

* build/generator/gen_win_dependencies.py

(GenDependenciesBase): s/15.0/14.1/

Patch by: atkawa7 <atkawa7{_AT_}yahoo.com>

This closes #6

Following up on r1803210, r1803127 and r1803261, move the declaration of

a few dependency related preprocessor defines to the dependency generator

instead of hardcoding them in the standard Windows configuration.

* build/generator/gen_win_dependencies.py

(_find_lz4,

_find_utf8proc): Add define here.

* subversion/svn_private_config.hw

(SVN_INTERNAL_LZ4,

SVN_INTERNAL_UTF8PROC): Remove define here.

  1. … 1 more file in changeset.
Add some plumbing for the Utf8proc and LZ4 dependencies on Windows.

This avoids receiving the following warnings when running gen-make.py:

Warning: Using undeclared dependency '$(SVN_UTF8PROC_LIBS)'.

Warning: Using undeclared dependency '$(SVN_LZ4_LIBS)'.

For now, we only support using the bundled versions of these libraries on

Windows, but this patch (minimally) integrates them into the build env.

* build/generator/gen_win_dependencies.py

(SVNCommonLibrary.__init__): Add new 'internal' field.

(GenDependenciesBase._find_lz4, GenDependenciesBase._find_utf8proc):

New functions that parse versions of bundled libraries and add them

as internal SVNCommonLibrary objects.

(GenDependenciesBase.find_libraries): Call new functions.

* build/generator/gen_win.py

(WinGeneratorBase.__init__): Separately print all internal libraries.

  1. … 1 more file in changeset.
* build/generator/gen_win_dependencies.py

(parse_options): Following up on r1786653, fix missed copy&paste error.

Add Microsoft Visual Studio 2017 support to our build scripting.

* gen-make.py

(_usage_exit): Document new support.

* build/generator/gen_win_dependencies.py

(parse_options): Add '2017' as explicitly handled option.

  1. … 1 more file in changeset.
Add support for default zlib filename (zlibstat.lib) for Windows builds which

is the default filename when using the ZLib's Visual Studio project file to

build ZLib.

* build/generator/gen_win_dependencies.py

(_find_zlib): Add check for zlibstat.lib in install-layout.

For Windows raise Apache httpd min version to 2.2.0 as per the minimal

supported httpd version for SVN >= 1.9.

* build/generator/gen_win_dependencies.py

(_find_httpd): raise minimal_httpd_version to 2.2.0

Obvious fix.

Rename the ill-defined SVN_LIBSVN_CLIENT_LINKS_* defines to an

easier to maintain SVN_LIBSVN_RA_LINKS_*. LIBSVN_CLIENT didn't

link any of this since well before 1.0.

* trunk/build/generator/gen_win_dependencies.py

* trunk/configure.ac

* trunk/subversion/libsvn_ra/ra_loader.c

* trunk/subversion/libsvn_ra_local/ra_plugin.c

* trunk/subversion/svn_private_config.hw

Update references.

  1. … 4 more files in changeset.
Following up on r1692093, also drop optional dependencies when fetching

the dependencies via a different route.

* build/generator/gen_win.py

(get_win_depends): Combine two code paths.

* build/generator/gen_win_dependencies.py

Stop setting unused self.bdb_lib.

  1. … 1 more file in changeset.
On Windows: drop BDB not found warning, if building without BDB-support.

This makes this code match the *nix configure code.

* build/generator/gen_win.py

(__init__): Remove BDB-warning, if optional 'db' library not found in

self._libraries

* build/generator/gen_win_dependencies.py

(parse_options): initialize self.bdb_path to None.

(_find_bdb): introduce local variable to determine bdb_path taking

either a specified path (via --with-berkeley-db) or

attempting the legacy default path ('db4-win32')

Only issue the warning, if failing to locate the BDB path

AND the user having explicitly specified the bdb-path.

Patch by: Stefan Hett <stefan{_AT_}egosoft.com>

  1. … 1 more file in changeset.
* build/generator/gen_win_dependencies.py

(_optional_libraries): Mark OpenSSL as optional and

move memcache in the detected dependencies.

Make Ruby bindings build on Windows with Visual Studio 2015.

* build/generator/gen_win_dependencies.py

(SVNCommonLibrary._find_ruby): Do not define snprintf for VS2015+ because

it's a proper function in that (and presumably newer) versions.

Fix detection of Ruby on Windows for Ruby 2.2+.

* build/generator/gen_win_dependencies.py

(SVNCommonLibrary._find_ruby): Use the RbConfig module, not the deprecated Config

module; this aligns Ruby detection on Windows with the way it's done in swig.m4.

* build/generator/gen_win_dependencies.py

(_find_apr_util_etc): Following up on r1674626, remove unneeded libname

definition. Users already depend on apr-util.

On Windows: enable detecting memcache support from apr-util 1.3 or later.

* build/generator/gen_win_dependencies.py

(find_libraries): Update caller.

(_find_apr_util_and_expat): Rename to...

(_find_apr_util_etc): ... this. Declare availability of memcache support

when we have apr-util 1.3 or later.

Add gen-make generator support for Visual Studio "2015"

* build/generator/gen_win_dependencies.py

Document support for VS 2015.

* gen-make.py

Add magic values for VS2015. Stop guessing that we have

VS 7.0 on future versions, as VS 2012 is a much better starting point

for future VS versions (as that has at least the latest .vcxproj file

support, which 7.0 didn't have.)

  1. … 1 more file in changeset.
Apply the standard pre-branch whitespace cleanup via

tools/dev/remove-trailing-whitespace.zsh

There should be no functional changes.

* .ycm_extra_conf.py

* build/generator/gen_win.py

* build/generator/gen_win_dependencies.py

* subversion/bindings/javahl/native/EditorProxy.h

* subversion/bindings/javahl/native/StateReporter.cpp

* subversion/bindings/javahl/native/org_apache_subversion_javahl_util_ConfigImpl_Category.cpp

* subversion/bindings/swig/perl/libsvn_swig_perl/swigutil_pl.c

* subversion/bindings/swig/perl/libsvn_swig_perl/swigutil_pl.h

* subversion/bindings/swig/python/libsvn_swig_py/swigutil_py.c

* subversion/bindings/swig/ruby/libsvn_swig_ruby/swigutil_rb.c

* subversion/bindings/swig/ruby/test/test_fs.rb

* subversion/bindings/swig/ruby/test/util.rb

* subversion/bindings/swig/ruby/test/windows_util.rb

* subversion/include/private/svn_fs_fs_private.h

* subversion/include/private/svn_packed_data.h

* subversion/include/private/svn_repos_private.h

* subversion/include/private/svn_sorts_private.h

* subversion/include/private/svn_subr_private.h

* subversion/include/private/svn_wc_private.h

* subversion/include/svn_auth.h

* subversion/include/svn_cache_config.h

* subversion/include/svn_client.h

* subversion/include/svn_cmdline.h

* subversion/include/svn_diff.h

* subversion/include/svn_fs.h

* subversion/include/svn_io.h

* subversion/include/svn_ra.h

* subversion/include/svn_repos.h

* subversion/include/svn_string.h

* subversion/libsvn_client/blame.c

* subversion/libsvn_client/commit.c

* subversion/libsvn_client/commit_util.c

* subversion/libsvn_client/copy.c

* subversion/libsvn_client/diff.c

* subversion/libsvn_client/externals.c

* subversion/libsvn_client/log.c

* subversion/libsvn_client/patch.c

* subversion/libsvn_client/ra.c

* subversion/libsvn_client/update.c

* subversion/libsvn_delta/svndiff.c

* subversion/libsvn_diff/parse-diff.c

* subversion/libsvn_fs/fs-loader.c

* subversion/libsvn_fs_base/lock.c

* subversion/libsvn_fs_fs/cached_data.c

* subversion/libsvn_fs_fs/fs_fs.c

* subversion/libsvn_fs_fs/hotcopy.c

* subversion/libsvn_fs_fs/id.c

* subversion/libsvn_fs_fs/index.c

* subversion/libsvn_fs_fs/index.h

* subversion/libsvn_fs_fs/lock.c

* subversion/libsvn_fs_fs/low_level.c

* subversion/libsvn_fs_fs/low_level.h

* subversion/libsvn_fs_fs/pack.c

* subversion/libsvn_fs_fs/recovery.c

* subversion/libsvn_fs_fs/rev_file.c

* subversion/libsvn_fs_fs/revprops.c

* subversion/libsvn_fs_fs/revprops.h

* subversion/libsvn_fs_fs/stats.c

* subversion/libsvn_fs_fs/transaction.c

* subversion/libsvn_fs_fs/tree.c

* subversion/libsvn_fs_fs/verify.c

* subversion/libsvn_fs_x/cached_data.c

* subversion/libsvn_fs_x/changes.c

* subversion/libsvn_fs_x/dag.h

* subversion/libsvn_fs_x/fs.h

* subversion/libsvn_fs_x/fs_id.c

* subversion/libsvn_fs_x/fs_x.c

* subversion/libsvn_fs_x/hotcopy.c

* subversion/libsvn_fs_x/hotcopy.h

* subversion/libsvn_fs_x/index.c

* subversion/libsvn_fs_x/index.h

* subversion/libsvn_fs_x/lock.c

* subversion/libsvn_fs_x/lock.h

* subversion/libsvn_fs_x/low_level.c

* subversion/libsvn_fs_x/low_level.h

* subversion/libsvn_fs_x/noderevs.c

* subversion/libsvn_fs_x/pack.c

* subversion/libsvn_fs_x/reps.c

* subversion/libsvn_fs_x/rev_file.c

* subversion/libsvn_fs_x/revprops.c

* subversion/libsvn_fs_x/revprops.h

* subversion/libsvn_fs_x/string_table.c

* subversion/libsvn_fs_x/string_table.h

* subversion/libsvn_fs_x/transaction.c

* subversion/libsvn_fs_x/tree.c

* subversion/libsvn_fs_x/util.c

* subversion/libsvn_fs_x/verify.c

* subversion/libsvn_ra_local/ra_plugin.c

* subversion/libsvn_ra_serf/ra_serf.h

* subversion/libsvn_ra_serf/serf.c

* subversion/libsvn_ra_serf/update.c

* subversion/libsvn_ra_serf/util.c

* subversion/libsvn_ra_svn/client.c

* subversion/libsvn_ra_svn/marshal.c

* subversion/libsvn_repos/authz_pool.c

* subversion/libsvn_repos/config_pool.c

* subversion/libsvn_repos/dump.c

* subversion/libsvn_repos/fs-wrap.c

* subversion/libsvn_repos/log.c

* subversion/libsvn_subr/bit_array.c

* subversion/libsvn_subr/cache-membuffer.c

* subversion/libsvn_subr/config.c

* subversion/libsvn_subr/dirent_uri.c

* subversion/libsvn_subr/fnv1a.c

* subversion/libsvn_subr/gpg_agent.c

* subversion/libsvn_subr/hash.c

* subversion/libsvn_subr/io.c

* subversion/libsvn_subr/object_pool.c

* subversion/libsvn_subr/packed_data.c

* subversion/libsvn_subr/prefix_string.c

* subversion/libsvn_subr/sorts.c

* subversion/libsvn_subr/sqlite.c

* subversion/libsvn_subr/subst.c

* subversion/libsvn_subr/sysinfo.c

* subversion/libsvn_subr/types.c

* subversion/libsvn_subr/utf.c

* subversion/libsvn_subr/utf8proc/utf8proc_data.c

* subversion/libsvn_subr/x509info.c

* subversion/libsvn_wc/diff.h

* subversion/libsvn_wc/props.c

* subversion/libsvn_wc/update_editor.c

* subversion/libsvn_wc/wc.h

* subversion/libsvn_wc/wc_db.c

* subversion/libsvn_wc/wc_db.h

* subversion/mod_dav_svn/lock.c

* subversion/mod_dav_svn/mod_dav_svn.c

* subversion/mod_dav_svn/status.c

* subversion/svn/auth-cmd.c

* subversion/svn/info-cmd.c

* subversion/svn/mergeinfo-cmd.c

* subversion/svn/notify.c

* subversion/svn/propget-cmd.c

* subversion/svn/svn.c

* subversion/svnadmin/svnadmin.c

* subversion/svnserve/logger.c

* subversion/svnserve/serve.c

* subversion/svnserve/server.h

* subversion/svnserve/svnserve.c

* subversion/tests/cmdline/blame_tests.py

* subversion/tests/cmdline/checkout_tests.py

* subversion/tests/cmdline/commit_tests.py

* subversion/tests/cmdline/export_tests.py

* subversion/tests/cmdline/externals_tests.py

* subversion/tests/cmdline/lock_tests.py

* subversion/tests/cmdline/merge_tests.py

* subversion/tests/cmdline/merge_tree_conflict_tests.py

* subversion/tests/cmdline/move_tests.py

* subversion/tests/cmdline/redirect_tests.py

* subversion/tests/cmdline/stat_tests.py

* subversion/tests/cmdline/svnadmin_tests.py

* subversion/tests/cmdline/switch_tests.py

* subversion/tests/cmdline/update_tests.py

* subversion/tests/libsvn_client/client-test.c

* subversion/tests/libsvn_client/mtcc-test.c

* subversion/tests/libsvn_fs/fs-test.c

* subversion/tests/libsvn_fs/locks-test.c

* subversion/tests/libsvn_fs_fs/fs-fs-fuzzy-test.c

* subversion/tests/libsvn_fs_fs/fs-fs-pack-test.c

* subversion/tests/libsvn_fs_x/fs-x-pack-test.c

* subversion/tests/libsvn_fs_x/string-table-test.c

* subversion/tests/libsvn_ra/ra-test.c

* subversion/tests/libsvn_subr/cache-test.c

* subversion/tests/libsvn_subr/checksum-test.c

* subversion/tests/libsvn_subr/config-test.c

* subversion/tests/libsvn_subr/io-test.c

* subversion/tests/libsvn_subr/packed-data-test.c

* subversion/tests/libsvn_subr/priority-queue-test.c

* subversion/tests/libsvn_subr/sqlite-test.c

* subversion/tests/libsvn_subr/string-test.c

* subversion/tests/libsvn_subr/subst_translate-test.c

* subversion/tests/libsvn_subr/x509-test.c

* subversion/tests/libsvn_wc/op-depth-test.c

* subversion/tests/svn_test_main.c

* tools/dev/benchmarks/RepoPerf/copy_repo.py

* tools/dev/benchmarks/RepoPerf/win_repo_bench.py

* tools/dev/build-svn-deps-win.pl

* tools/dev/fsfs-access-map.c

* tools/dev/po-merge.py

* tools/dev/x509-parser.c

* tools/diff/diff.c

* tools/dist/backport.pl

* tools/dist/backport_tests.py

* tools/hook-scripts/mailer/mailer.py

* tools/server-side/svnpubsub/daemonize.py

* win-tests.py

Remove unneeded whitespace via script.

  1. … 184 more files in changeset.
* generator/gen_win_dependencies.py

(_find_openssl): Update regex to support OpenSSL's new formatting rules.

Correctly handle a situation of an unresolved SQLite dependency in

our gen_win_dependencies.py script, instead of printing a stacktrace

(TypeError: not all arguments converted during string formatting).

* build/generator/gen_win_dependencies.py

(_find_sqlite): The error message does not have any '%s' placeholders,

so there is no need to pass the self.sqlite_path format argument.

* build/generator/gen_win_dependencies.py

(_find_perl): Following up on r1617030, properly read the new line instead

of using the same line twice.

On Windows: Properly detect the right name of the perl library file on

Windows, for newer perl versions that sometimes generate a .lib file

with the more linux like libperl158.a name.

* build/generator/gen_win_dependencies.py

(_find_perl): Ask perl for the library name instead of guessing one

ourselves.

Fix another problem in the gen_win_dependencies.py script. There are some

cases when we want to print a warning about an unresolved dependency, e.g.:

[[[

WARNING: 'opensslv.h' not found

Use '--with-openssl' to configure openssl location.

]]]

Prior to this changeset, we would instead end up with printing a stacktrace

("... global name 'show_warning' is not defined") instead of issuing a

warning and continuing to work. The reason for that is somewhat trivial --

we inconsistently use the names 'show_warning' and 'show_warnings' across

the script and sometimes encounter a situation when the name of the function

argument does not match the name of the variable in its body. Fix this by

using a consistent naming scheme across the whole script file.

* build/generator/gen_win_dependencies.py

(_find_httpd, _find_openssl, _find_jdk, _find_swig, _find_serf,

_find_sasl, _find_libintl): Use 'show_warnings' instead of 'show_warning'

as the argument names. The discrepancies are only in the openssl,

jdk, and swig-related functions, but using a consistent naming scheme

*everywhere* could probably avoid the same mistakes in the future.

Do not check whether the apu_version.h file exists in the supplied APR-Util

location *twice*. os.path.isfile(path) implicates os.path.exists(path), so

the second check effectively does nothing.

* build/generator/gen_win_dependencies.py

(_find_apr_util_and_expat): Only check for the existence of the

'apu_version.h' once.

Following up on r1504501, gracefully exit in situations when the APR include

is missing instead of throwing out a cryptic stacktrace ("... local variable

'version_file_path' referenced before assignment").

* build/generator/gen_win_dependencies.py

(_find_apr): Do not attempt to use the uninitialized 'version_file_path'

variable in the error message, because we initialize it later. Just

stick to the same approach as in the _find_apr_util_and_expat(), i.e.

use the hardcoded filename in the error message.

Create Visual Studio 2010+ compatible solutions and projects when generating

projects for VS 2012 and later. This introduces/fixes support for the Visual

Studio '14' compile chain's projects via the 'fallback' 14 behavior.

As the current estimated release time for VS '14' is 2015 at earliest, I'm

not adding a more specific name yet.

* build/generator/gen_win_dependencies.py

Make regex matches for deprecated versions a bit stricter. Fix vsversion

typo.

* build/generator/templates/vcnet_sln.ezt

Generate compatible versions header, specifying compatibility with VS 2010

SP 1 (which introduced support for the 12.0 format, really added in VS2012).

  1. … 1 more file in changeset.
Remove some (since at least 1.8.0) unused gen-make options, from when we only

supported in-tree builds of bdb and apr-util.

* gen-make.py

Remove support for --enable-bdb-in-apr-util,

--with-neon and --without-neon.

* build/generator/gen_win_dependencies.py

Stop parsing enable-bdb-in-apr-util.

  1. … 1 more file in changeset.