Parts of the testsuite are used by third party packages. Even if you don't want to install test modules, you should install those.
This is what I identify as part of the test framework:

           $(scriptdir)/test/{libregrtest,support} \
            $(scriptdir)/test/{ann_module,ann_module2,ann_module3}.py \
            $(scriptdir)/test/{regrtest,test_support,__init__,__main__}.py

However that list might be incomplete.

For packaging purposes I think it's better to remove those files which you don't want to ship after the installation.

@doko42 Thanks for your review. After I installed the Python3.8 distribution for Ubuntu, it indeed has the regression test framework shipped but with all other test stuffs removed. So always shipping test framework skeleton regardless of the option value of --enable-test-modules could be a good idea. Could we do the same thing like Ubuntu does as shown below?

Under the directory /usr/lib/python3.8/test, Ubuntu ships:
__init__.py libregrtest __main__.py regrtest.py support test_support.py

In addition, Ubuntu doesn't ship test/{ann_module,ann_module2,ann_module3}.py. How are these used by 3rd-party?

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

The buildroot project has the following patch on Python which adds --disable-test-modules configure option:
https://git.buildroot.org/buildroot/tree/package/python3/0013-Add-an-option-to-disable-installation-of-test-module.patch

https://bugs.debian.org/944303 showed that python-typing-extensions uses the ann* modules. Now that's integrated in upstream cpython, but still shipped on PyPi: https://pypi.org/project/typing-extensions/. So yes, I think these could be removed as well.

Please note that Debian/Ubuntu packaging doesn't remove any of these files, but just ships them in a separate binary package, e.g. libpython3.8-testsuite.

@vstinner Regarding the regression test framework(see below for files and directories) what is your opinion? Keeping it or removing it entirely when disabling test suites? Looks buildroot removes it while Ubuntu keeps it. If we decide keeping it, some of the test extension modules(_testcapi and _testinternalcapi) have to be kept as well because test.support needs to import them.

__init__.py libregrtest __main__.py regrtest.py support test_support.py

I have made the requested changes; please review again

Thanks for making the requested changes!

@vstinner: please review the changes made to this pull request.

The Fedora package put the following files/directories in the python3-test binary RPM:

%if %{without flatpackage}
%files -n %{pkgname}-test
%endif

%{pylibdir}/ctypes/test
%{pylibdir}/distutils/tests
%{pylibdir}/sqlite3/test
%{pylibdir}/test
%{dynload_dir}/_ctypes_test.%{SOABI_optimized}.so
%{dynload_dir}/_testbuffer.%{SOABI_optimized}.so
%{dynload_dir}/_testcapi.%{SOABI_optimized}.so
%{dynload_dir}/_testimportmultiple.%{SOABI_optimized}.so
%{dynload_dir}/_testinternalcapi.%{SOABI_optimized}.so
%{dynload_dir}/_testmultiphase.%{SOABI_optimized}.so
%{dynload_dir}/_xxtestfuzz.%{SOABI_optimized}.so
%{pylibdir}/lib2to3/tests
%{pylibdir}/tkinter/test
%{pylibdir}/unittest/test

https://src.fedoraproject.org/rpms/python3.10/blob/master/f/python3.10.spec#_1429

With the PR and --disable-test-modules, the following directories (and sub-directories) are no longer installed:

ctypes/test/
distutils/tests/
idlelib/idle_test/
lib2to3/tests/
sqlite3/test/
test/
tkinter/test/
unittest/test/

With the PR and --disable-test-modules, the following files of lib-dynload/ are no longer installed:

_ctypes_test.cpython-310-x86_64-linux-gnu.so
_testbuffer.cpython-310-x86_64-linux-gnu.so
_testcapi.cpython-310-x86_64-linux-gnu.so
_testimportmultiple.cpython-310-x86_64-linux-gnu.so
_testinternalcapi.cpython-310-x86_64-linux-gnu.so
_testmultiphase.cpython-310-x86_64-linux-gnu.so
_xxtestfuzz.cpython-310-x86_64-linux-gnu.so

Install in /opt/py310 without tests:

./configure --disable-test-modules --prefix /opt/py310test CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 

Install in /opt/py310test with tests:

./configure --prefix /opt/py310test CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 

Remove .pyc files:

find /opt/py310 -name "*.pyc" -delete
find /opt/py310test -name "*.pyc" -delete

Compare the content of the two directories:

(cd /opt/py310test/lib64/python3.10/; find)|sort > A
(cd /opt/py310/lib64/python3.10/; find)|sort > B
diff -u B A > pr23886_files.diff

I atttached the full pr23886_files.diff to https://bugs.python.org/issue27640

Good: this PR does not remove any file/directory by default (compared to the master branch). It works as expected.

$ git switch master
$ ./configure --prefix /opt/py310master CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 
$ find /opt/py310master -name "*.pyc" -delete

$ git switch pr/23886
$ ./configure --prefix /opt/py310pr CFLAGS="-O0" --with-platlibdir=lib64 && make && make install 
$ find /opt/py310pr -name "*.pyc" -delete

$ (cd /opt/py310master/; find)|sort > A; (cd /opt/py310pr/; find)|sort > B
$ diff -u A B

# diff output is empty: that's good! there is no change

On Fedora, the python3 package does not contain the test module. You must install python3-test.

The test module should not be used by third party projects. The documentation starts with the note:

The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

https://docs.python.org/dev/library/test.html

@doko42:

https://bugs.debian.org/944303 showed that python-typing-extensions uses the ann* modules. Now that's integrated in upstream cpython, but still shipped on PyPi: https://pypi.org/project/typing-extensions/. So yes, I think these could be removed as well.

Please note that Debian/Ubuntu packaging doesn't remove any of these files, but just ships them in a separate binary package, e.g. libpython3.8-testsuite.

On Debian, Ubuntu or Fedora, it's just a matter of putting the right dependency on the libpython3.8-testsuite (python3-test on Fedora) package.

@pxinwr:

@vstinner Regarding the regression test framework(see below for files and directories) what is your opinion? Keeping it or removing it entirely when disabling test suites? Looks buildroot removes it while Ubuntu keeps it.

This PR is not intended for general Linux distributions like Debian or Fedora, but specialized embedded devices. Xavier created https://bugs.python.org/issue27640 when he worked on putting Python on Android. The buildroot is also intended to be used to produce an image for an embedded devices. The use case is to ship an image for end users who will not run any test but use a "finished" product. For developers, well, just don't use --disable-test-modules.

I tested again the PR, at commit 9e5dada. I rebased locally the PR on master. The PR works as expected!

Install Python 3 times:

• master
• PR with tests
• PR without tests

# master with tests
$ git switch master
$ git clean -fdx
$ ./configure --prefix /opt/py310_master CFLAGS="-O0"
$ make && make install

# PR with tests
$ git switch pr/23886
$ git clean -fdx
$ ./configure --disable-test-modules --prefix /opt/py310_notest CFLAGS="-O0"
$ make && make install

# PR without tests
$ git switch pr/23886
$ git clean -fdx
$ ./configure --prefix /opt/py310_pr CFLAGS="-O0"
$ make && make install

First test: ensure that the PR still copy exactly the same files: good, there is zero difference!

$ (cd /opt/py310_master; find|sort) > files-master
$ (cd /opt/py310_pr; find|sort) > files-pr-with-tests
$ diff -u files-master files-pr-with-tests 

# diff output is empty: good! there is no change

Second test: check that --disable-test-modules don't install tests. Good! Test C extension are not built nor installed and test subdirectories are not installed, as expected.

$ (cd /opt/py310_master; find|sort) > files-master
$ (cd /opt/py310_notest; find|sort) > files-pr-without-tests

$ (cd /opt/py310_master; find -type d|sort) > dirs-master
$ (cd /opt/py310_notest; find -type d|sort) > dirs-pr-without-tests

# Removed files
$ diff -u files-master files-pr-without-tests |grep '^-.*so$'
-./lib/python3.10/lib-dynload/_ctypes_test.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testbuffer.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testcapi.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testimportmultiple.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testinternalcapi.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_testmultiphase.cpython-310-x86_64-linux-gnu.so
-./lib/python3.10/lib-dynload/_xxtestfuzz.cpython-310-x86_64-linux-gnu.so

# Removed directories
$ diff -u dirs-master dirs-pr-without-tests |grep ^-|grep -v __pycache__
-./lib/python3.10/ctypes/test
-./lib/python3.10/distutils/tests
-./lib/python3.10/idlelib/idle_test
-./lib/python3.10/lib2to3/tests
-./lib/python3.10/lib2to3/tests/data
-./lib/python3.10/lib2to3/tests/data/fixers
-./lib/python3.10/lib2to3/tests/data/fixers/myfixes
-./lib/python3.10/sqlite3/test
-./lib/python3.10/test
-./lib/python3.10/test/audiodata
-./lib/python3.10/test/capath
-./lib/python3.10/test/cjkencodings
-./lib/python3.10/test/data
-./lib/python3.10/test/decimaltestdata
-./lib/python3.10/test/dtracedata
-./lib/python3.10/test/encoded_modules
-./lib/python3.10/test/imghdrdata
-./lib/python3.10/test/libregrtest
-./lib/python3.10/test/sndhdrdata
-./lib/python3.10/test/subprocessdata
-./lib/python3.10/test/support
-./lib/python3.10/test/test_asyncio
-./lib/python3.10/test/test_email
-./lib/python3.10/test/test_email/data
-./lib/python3.10/test/test_import
-./lib/python3.10/test/test_import/data
-./lib/python3.10/test/test_import/data/circular_imports
-./lib/python3.10/test/test_import/data/circular_imports/subpkg
-./lib/python3.10/test/test_import/data/package
-./lib/python3.10/test/test_import/data/package2
-./lib/python3.10/test/test_import/data/unwritable
-./lib/python3.10/test/test_importlib
-./lib/python3.10/test/test_importlib/builtin
-./lib/python3.10/test/test_importlib/data
-./lib/python3.10/test/test_importlib/data01
-./lib/python3.10/test/test_importlib/data01/subdirectory
-./lib/python3.10/test/test_importlib/data02
-./lib/python3.10/test/test_importlib/data02/one
-./lib/python3.10/test/test_importlib/data02/two
-./lib/python3.10/test/test_importlib/data03
-./lib/python3.10/test/test_importlib/data03/namespace
-./lib/python3.10/test/test_importlib/data03/namespace/portion1
-./lib/python3.10/test/test_importlib/data03/namespace/portion2
-./lib/python3.10/test/test_importlib/extension
-./lib/python3.10/test/test_importlib/frozen
-./lib/python3.10/test/test_importlib/import_
-./lib/python3.10/test/test_importlib/namespace_pkgs
-./lib/python3.10/test/test_importlib/namespace_pkgs/both_portions
-./lib/python3.10/test/test_importlib/namespace_pkgs/both_portions/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/module_and_namespace_package
-./lib/python3.10/test/test_importlib/namespace_pkgs/module_and_namespace_package/a_test
-./lib/python3.10/test/test_importlib/namespace_pkgs/not_a_namespace_pkg
-./lib/python3.10/test/test_importlib/namespace_pkgs/not_a_namespace_pkg/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion1
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion1/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion2
-./lib/python3.10/test/test_importlib/namespace_pkgs/portion2/foo
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project1/parent/child
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project2/parent/child
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3/parent
-./lib/python3.10/test/test_importlib/namespace_pkgs/project3/parent/child
-./lib/python3.10/test/test_importlib/source
-./lib/python3.10/test/test_importlib/zipdata01
-./lib/python3.10/test/test_importlib/zipdata02
-./lib/python3.10/test/test_json
-./lib/python3.10/test/test_peg_generator
-./lib/python3.10/test/test_tools
-./lib/python3.10/test/test_warnings
-./lib/python3.10/test/test_warnings/data
-./lib/python3.10/test/test_zoneinfo
-./lib/python3.10/test/test_zoneinfo/data
-./lib/python3.10/test/tracedmodules
-./lib/python3.10/test/xmltestdata
-./lib/python3.10/test/xmltestdata/c14n-20
-./lib/python3.10/test/ziptestdata
-./lib/python3.10/tkinter/test
-./lib/python3.10/tkinter/test/test_tkinter
-./lib/python3.10/tkinter/test/test_ttk
-./lib/python3.10/unittest/test
-./lib/python3.10/unittest/test/testmock

Thank you @pxinwr for your multiple updates, the final PR is much better. The change is now properly documented (NEWS entry, What's New in Python 3.10 entry), the option is properly documented directly in configure (it says exactly what it does), it's better than Xavier's initial patch and the Thoma's buildbroot patch since it also doesn't build test extensions. And overall, I think that it's a reasonable build option, it makes sense to not install tests on a small "embedded device".

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora 3.x has failed when building commit 277ce30.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/223/builds/528) and take a look at the build logs.
4. Check if the failure is related to this commit (277ce30) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/223/builds/528

Failed tests:

• test_asyncio

Failed subtests:

• test_start_new_session - test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests

Summary of the results of the build (if available):

==

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/asyncio/unix_events.py", line 1302, in _do_waitpid
    loop, callback, args = self._callbacks.pop(pid)
KeyError: 3703654


Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z/build/Lib/test/test_asyncio/test_subprocess.py", line 169, in test_start_new_session
    self.assertEqual(exitcode, 8)
AssertionError: 255 != 8