Checkout
troycurtisjr
committed
on 27 Dec 18
On branch swig-py3: Use bytes for all API string parameters.

Since many string like arguments in the API can contain arbitrary bytes, use t… Show more
On branch swig-py3: Use bytes for all API string parameters.

Since many string like arguments in the API can contain arbitrary bytes, use the

Python Bytes interface in Python 3, while still using Str in Python 2, to ensure

full compatibility with all possible input data.

Build system change:

* build/ac-macros/swig.m4:

 Add "-DPY3" option to SWIG_PY_OPTS for py3 build to define symbol to switch

 typemap between py2 and py3.

Typemap changes:

* subversion/bindings/swig/core.i

 (%typemap(argout) (char *buffer, apr_size_t *len):

   Map buffer arg to Bytes insted of Str.

 (%typemap(in) (const char *data, apr_size_t *len) ($*2_type temp):

   Map data arg to Bytes instead of Str and pass Py_ssize_t variable as length

   argment of PyBytes_AsStringAndSize for for explicit type conversion.

 (%typemap(in) const void *value):

   Map value arg to Bytes instead of Str for svn_auth_set_parameter().

* subversion/bindings/swig/include/svn_global.swg:

 - Define SWIG_PYTHON_STRICT_BYTE_CHAR symbol for swig context for

   Python3.

 - Switch "in" type typemap for char *, char const *, char * const,

   and char const * const "", to use parse parameter "y" for py3 and

   "s" for py2.

* subversion/bindings/swig/include/svn_string.swg

 (%typemap(argout) RET_STRING, %typemap(in) svn_stringbuf_t *,

   %typemap(out) svn_stringbuf_t *, %typemap(in) const svn_string_t *,

   %typemap(out) svn_string_t *, %typemap(argout) const char **OUTPUT):

   Map args to Bytes instead of Str.

* subversion/bindings/swig/include/svn_swigcompat.swg

 (%set_constant(name, value): Use PyDict_SetItem and PyBytes_FromString

   instead of PyDict_SetItemString, for SWIG <= 1.3.27 (not tested).

* subversion/bindings/swig/include/svn_types.swg

 (%typemap(in) const char *MAY_BE_NULL): Don't use "z" conversion

   format and convert to Bytes if it is not NULL.

 (%typemap(in) (const char *PTR, apr_size_t LEN): Use PyBytes_Check and

   PyBytes_AsStringAndSize instead of PyStr_Check and PyStr_AsUTF8AndSize.

* subversion/bindings/swig/python/libsvn_swig_py/swigutil_py.c

 (make_string_from_ob(), make_svn_string_from_ob(),

  svn_swig_py_make_file(), svn_swig_py_get_commit_log_func()):

   Use PyBytes_Check and PyBytes_AsString instead of PyStr_Check and

   PyStr_AsString.

 (convert_hash()): Use Dict_SetItem and PyBytes_FromString instead of

   DictItemString.

 (convert_svn_string_t(), svn_swig_py_proparray_to_dict(),

  ra_callbacks_get_wc_prop()):

   Use PyBytes_AsStringAndSize instead of PyStr_AsUTF8AndSize.

  (cstring_to_pystring(), convert_string(),

   svn_swig_py_propinheriteditemarray_to_dict(),

   svn_swig_py_proparray_to_dict(),

   svn_swig_py_locationhash_to_dict(),

   svn_swig_py_c_strings_to_list(),

   svn_swig_py_changed_path_hash_to_dict(),

   svn_swig_py_changed_path2_hash_to_dict(),

   svn_swig_py_unwrap_string(),

   svn_swig_py_array_to_list()): Use PyBytes_FromString instead of

    PyStr_FromString.

 (delete_entry(), add_directory(), open_directory(), change_dir_prop(),

   add_file(), open_file(), apply_textdelta(), change_file_prop(),

   close_file(), parse_fn3_uuid_record(), parse_fn3_set_revision_property(),

   parse_fn3_set_node_property(), parse_fn3_delete_node_property(),

   svn_swig_py_notify_func(), svn_swig_py_status_func(),

   svn_swig_py_delta_path_driver_cb_func(), svn_swig_py_status_func2(),

   svn_swig_py_fs_lock_callback(), svn_swig_py_repos_authz_func(),

   svn_swig_py_repos_history_func(), svn_swig_py_proplist_receiver2(),

   svn_swig_py_log_receiver(), svn_swig_py_info_receiver_func(),

   svn_swig_py_client_blame_receiver_func(),

   svn_swig_py_changelist_receiver_func(),

   svn_swig_py_auth_gnome_keyring_unlock_prompt_func(),

   svn_swig_py_auth_simple_prompt_func(),

   svn_swig_py_auth_username_prompt_func(),

   svn_swig_py_auth_ssl_server_trust_prompt_func(),

   svn_swig_py_auth_ssl_client_cert_prompt_func(),

   svn_swig_py_auth_ssl_client_cert_pw_prompt_func(),

   svn_swig_py_config_auth_walk_func(), ra_callbacks_get_wc_prop(),

   ra_callbacks_push_or_set_wc_prop(), ra_callbacks_invalidate_wc_props(),

   svn_swig_py_commit_callback(), svn_swig_py_ra_file_rev_handler_func(),

   svn_swig_py_ra_lock_callback(), reporter_set_path(),

   reporter_delete_path(), reporter_link_path(),

   wc_diff_callbacks2_file_changed_or_added(),

   wc_diff_callbacks2_file_deleted(), wc_diff_callbacks2_dir_added(),

   wc_diff_callbacks2_dir_deleted(),

   wc_diff_callbacks2_dir_props_changed(),

   svn_swig_py_config_enumerator2(),

   svn_swig_py_config_section_enumerator2()): Use 's' for py2 and use 'y'

    for py3 in argment format string.

 (ra_callbacks_push_or_set_wc_prop()): Use PyBytes_FromStringAndSize

   instead of PyStr_FromStringAndSize.

 (ra_callbacks_get_client_string()): Use PyBytes_FromString instaed of

   PyStr_AsString.

* subversion/bindings/swig/svn_client.i

 (%typemap(argout) apr_array_header_t **props):

  Use PyBytes_FromStringAndSize() instead of PyStr_FromStringAndSize.

Helper python code changes:

* subversion/bindings/swig/python/svn/core.py

 (svn_path_compare_paths): Do not use cmp, for py3 compatibility.

 (Stream.read()): Chunks is now list of bytes, not list of str.

* subversion/bindings/swig/python/svn/fs.py

 (FileDiff.get_pipe()):

  - Make sure self.tempfile1 and self.self.tempfile2 are bytes.

  - Pass bytes to header_encoding arg in _svndiff.file_output_unified4().

* subversion/bindings/swig/python/svn/ra.py

 (Callbacks..__doc__):

   Treat path as bytes in sample code.

* subversion/bindings/swig/python/svn/repos.py

 (ChangeCollector._make_base_path()): Treat path as bytes.

 (ChangeCollector.open_root(): Path and basepath should be bytes in

   dir_baton.

 (RevisionChangeCollector._make_base_path()): Treat path as bytes.

Unit test changes:

* subversion/bindings/swig/python/tests/auth.py:

 Replace all str literals with bytes literals except docstring and module name.

* subversion/bindings/swig/python/tests/checksum.py

 (ChecksumTestCases.test_checksum()): Check if type of check_val is

  bytes instead of str.

* subversion/bindings/swig/python/tests/client.py:

 Replace all str literals with bytes literals except for the mode arg of open(),

   arguments to utils.Temper methods, docstrings, and module names.

 (SubversionClientTestCase.test_uuid_from_url): Check if return value

   type client.uuid_from_url() is bytes instead of str.

 (SubversionClientTestCase.test_uuid_from_path): Check if return value

   type client.uuid_from_path() is bytes instead of str.

 (SubversionClientTestCase.test_merge_peg3()): Open the result file in

   raw mode.

 (SubversionClientTestCase.test_update4()): Convert os.path.sep into

   bytes if it is str in py3.

* subversion/bindings/swig/python/tests/core.py:

 Replace all str literals with bytes literals except for exception messages,

   arguments to utils.Temper methods, docstrigs, and module names.

 Add new tests for svn_stream_*()

 (SubversionCoreTestCase.test_stream_from_stringbuf()): New test method.

 (SubversionCoreTestCase.test_stream_read_full()): New test method.

 (SubversionCoreTestCase.test_stream_read2()): New test method.

 (SubversionCoreTestCase.test_stream_write_exception()): New test method.

 (SubversionCoreTestCase.test_stream_write()): New test method.

 (SubversionCoreTestCase.test_stream_readline()): New test method.

* subversion/bindings/swig/python/tests/delta.py

(DeltaTestCase.estTxWindowHandler_stream_IF(),

 DeltaTestCase.estTxWindowHandler_Stream_IF()):

 Use bytes literals to make stream when using

  svn.core.svn_stream_from_stringbuf().

 Use bytes file name to make stream when using

  svn.core.svn_stream_from_aprfile2().

* subversion/bindings/swig/python/tests/fs.py

 (SubversionFSTestCase.log_message_func(): log_msg_func3 callback now

   expects bytes to return.

 (SubversionFSTestCase.setUp()): Pass bytes path and url to client.import2().

 (SubversionFSTestCase.test_diff_repos_paths_internal(),

  SubversionFSTestCase.test_diff_repos_paths_external()): Use bytes for

   path1 argument to fs.FileDiff.__init__().

* subversion/bindings/swig/python/tests/mergeinfo.py

 (SubversionMergeinfoTestCase.TEXT_MERGEINFO1,

  SubversionMergeinfoTestCase.TEXT_MERGEINFO2,

  SubversionMergeinfoTestCase.MERGEINFO_SRC): Use bytes instead of str.

 (SubversionMergeinfoTestCase.test_mergeinfo_get()):

  - Pass list of bytes for paths arg to repos.fs_get_mergeinfo().

  - Replace str in expected_mergeinfo with bytes.

* subversion/bindings/swig/python/tests/ra.py

 Replace all str literals with bytes literals except assertion message,

 arguments to utils.Temper methods, docstrings, and module names.

 (SubversionRepositoryAccessTestCase.test_get_file): Compare file

  content as bytes.

* subversion/bindings/swig/python/tests/repository.py

 Replace all str literals with bytes literals when passed into or returned from

   the Subversion API, except exception messages.

 (SubversionRepositoryTestCase.test_dump_fs2()): Treat dump and

  feedback as bytes and compose expected_feedback as bytes.

* subversion/bindings/swig/python/tests/utils.py

 (Temper.alloc_empty_dir(),

  Temper.alloc_empty_repo): Convert return value of tempfile.mkdtemp() into

   bytes.

 (Temper.file_uri_for_path()): Return URI as bytes.

* subversion/bindings/swig/python/tests/wc.py

 Replace all str literals with bytes literals when passed into or returned from

   the Subversion API, except exception messages.

 (SubversionWorkingCopyTestCase.test_get_adm_dir()): Check type as

   bytes.

 (SubversionWorkingCopyTestCase.test_get_pristine_copy_path(),

  SubversionWorkingCopyTestCase.test_diff_editor4()): Open file in raw mode.

 (SubversionWorkingCopyTestCase.test_commit()): Use bytes literals

   instead of str.

* subversion/bindings/swig/python/tests/trac/versioncontrol/main.py

 (Node.DIRECTORY, Node.FILE): Replace str with bytes.

 (Node.__init__): Do not convert path to str.

 (Node.get_name): Treat self.path as bytes.

 (Changeset.ADD, Changeset.COPY, Changeset.DELETE, Changeset.EDIT,

  Changeset.MOVE): Replace str with bytes.

* subversion/bindings/swig/python/tests/trac/versioncontrol/svn_fs.py

 (SubversionRepository.__init__(),

  SubversionRepository.get_oldest_rev(),

  SubversionRepository.youngest_rev(),

  SubversionRepository.next_rev()): Treat self.path and self.scope as bytes.

 (SubversionRepository.normalize_path(),

  SubversionRepository.get_node()): Treat path as bytes.

 (SubversionRepository.get_path_history()): Replace 'unknown' Changeset value

   with bytes.

 (SubversionRepository.get_deltas()): Replace str 'entry' arg to

   repos.svn_repos_dir_delta() with bytes.

 (SubversionNode.__init__()):

  - Treat self.path and self.scope as bytes.

  - Decode path into str for exception message on py3.

 (SubversionNode.get_entries()): Treat self.path as bytes.

 (SubversionNode.get_properties()): Property values are bytes, not str.

 (DiffChangeEditor.open_root()): Return bytes value.

* subversion/bindings/swig/python/tests/trac/versioncontrol/tests/svn_fs.py

 Replace all str literals with bytes literals when passed into or returned from

   the Subversion API, except exception messages.

 (REPOS_URL) Build REPOS_URL as bytes.

Patch By: Yasuhito FUTATSUKI <futatuki@poem.co.jp>

Show less