bpo-45950: Introduce Bootstrap Python again #29859

tiran · 2021-11-30T09:56:12Z

The build system now uses a :program:_bootstrap_python interpreter for
freezing and deepfreezing again. To speed up build process the build tools
:program:_bootstrap_python and :program:_freeze_module are no longer
build with LTO.

Cross building depends on a build Python interpreter, which must have same
version and bytecode as target host Python.

https://bugs.python.org/issue45950

Programs/_bootstrap_python.c

bedevere-bot · 2021-11-30T10:14:22Z

🤖 New build scheduled with the buildbot fleet by @tiran for commit b757d5bd526cfffeb134eb6744ac86c561e61e9d 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

bedevere-bot · 2021-11-30T12:47:15Z

🤖 New build scheduled with the buildbot fleet by @tiran for commit fa4cd7fc9f3763ff1309d8bfbc97853223f17239 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

gvanrossum · 2021-11-30T18:09:40Z

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

vstinner · 2021-11-30T19:05:12Z

it's just an extra link step

When LTO is used, a link step can be quite slow. I noticed that when there was a second _bootstrap_python binary. I didn't check if _testembed has a similar issue or not. I'm mostly developing Python without --enable-shared: Python is linked "statically" in all built executables.

vstinner · 2021-11-30T19:05:32Z

Maybe LTO flags can be skipped on these less important binaries.

vstinner · 2021-11-30T21:00:24Z

FYI the revert #29717 fixed an issue with out of tree build. Please try to make sure that this bug is not reintroduced: https://bugs.python.org/issue45866#msg406764

tiran · 2021-12-01T08:50:54Z

Why not just always use _bootstrap_python, like my original PR? That way that code path is properly tested (not depending on a buildbot) and you don't need the extra conditional logic in configure. The build-time overhead is pretty small, all the .o files that go into it have to be built anyway, so it's just an extra link step.

We need some extra logic in configure.ac and Makefile anyway. The _bootstrap_python approach is incompatible with cross compiling. When you cross compile for a different architecture or platform, then any compiled code is not executable on the build host.

The approach also speeds up local development as long as you have a recent Python installation on your computer. With _bootstrap_python almost every code file gets rebuild whenever a core file changes. Scratch that, the Program/_freeze_module dependency triggers a partial rebuild.

I'll come up with a better way to test _buildbot_python.

gvanrossum

I am in favor of this. Thanks for doing this!

hroncok · 2021-12-02T13:53:13Z

I'll try to build this as RPM in Fedora to see if there are some problems with our bootstrapped OOT build.

hroncok · 2021-12-02T15:04:54Z

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

...
gcc -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g   -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now    -g  -fno-lto -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o   Python/pydtrace.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o -ldl                         -lm 
/usr/bin/ld: Programs/_freeze_module.o: in function `runtime_init':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:44: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o: in function `_Py_DECREF':
/builddir/build/BUILD/cpython-bootstrap_again/Include/object.h:493: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:(.data.rel.ro+0x0): undefined reference to `__gcov_indirect_call'
/usr/bin/ld: Programs/_freeze_module.o: in function `main':
/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:191: undefined reference to `__gcov_indirect_call'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_indirect_call_profiler_v4'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:233: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:211: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: /builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: undefined reference to `__gcov_time_profiler_counter'
/usr/bin/ld: Programs/_freeze_module.o:/builddir/build/BUILD/cpython-bootstrap_again/Programs/_freeze_module.c:218: more undefined references to `__gcov_time_profiler_counter' follow
...

tiran · 2021-12-02T15:29:12Z

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

tiran · 2021-12-02T15:55:04Z

OOT build with optimizations is passing for me on F35:

../../configure -C --enable-optimizations``
make

bedevere-bot · 2021-12-02T16:22:36Z

🤖 New build scheduled with the buildbot fleet by @tiran for commit 02da340e62db33a12aad1b03adf83a488e0255fb 🤖

If you want to schedule another build, you need to add the "🔨 test-with-buildbots" label again.

hroncok · 2021-12-02T16:30:46Z

I am getting a lot of undefined references to __gcov_* stuff, not sure if actually caused by this PR:

Please try again. I have rebased the PR and added LDFLAGS_NODIST to freeze and bootstrap.

That helped.

tiran · 2021-12-03T11:05:32Z

Rebased on top of #29902

The build system now uses a `_bootstrap_python` interpreter for freezing and deepfreezing again. Cross building depends on a build Python interpreter. `_bootstrap_python` and `_freeze_module` are no longer compiled with LTO. Link time optimization is very slow.

bedevere-bot · 2021-12-03T15:01:14Z

@tiran: Please replace # with GH- in the commit message next time. Thanks!

the-knights-who-say-ni added the CLA signed label Nov 30, 2021

bedevere-bot added the awaiting core review label Nov 30, 2021

tiran force-pushed the bootstrap_again branch from fe57850 to b757d5b Compare Nov 30, 2021

vstinner reviewed Nov 30, 2021

View changes

Programs/_bootstrap_python.c Outdated Show resolved Hide resolved

Programs/_bootstrap_python.c Outdated Show resolved Hide resolved

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

tiran force-pushed the bootstrap_again branch from b757d5b to fa4cd7f Compare Nov 30, 2021

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 30, 2021

gvanrossum self-requested a review Nov 30, 2021

tiran force-pushed the bootstrap_again branch 3 times, most recently from 757a652 to 551fb30 Compare Dec 1, 2021

gvanrossum approved these changes Dec 1, 2021

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Dec 1, 2021

tiran force-pushed the bootstrap_again branch from 551fb30 to cfe4551 Compare Dec 1, 2021

tiran changed the title ~~Bootstrap Python again~~ bpo-45950: Introduce Bootstrap Python again Dec 1, 2021

tiran marked this pull request as ready for review Dec 1, 2021

tiran force-pushed the bootstrap_again branch from cfe4551 to 02da340 Compare Dec 2, 2021

tiran added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Dec 2, 2021

tiran mentioned this pull request Dec 2, 2021

bpo-45949: Pure Python freeze module for cross builds (GH-29899) #29899

Merged

tiran force-pushed the bootstrap_again branch from 02da340 to 252b53a Compare Dec 3, 2021

tiran requested a review from a team as a code owner Dec 3, 2021

tiran force-pushed the bootstrap_again branch from 252b53a to c630ffd Compare Dec 3, 2021

tiran force-pushed the bootstrap_again branch from c630ffd to ba35cc4 Compare Dec 3, 2021

tiran merged commit 84ca123 into python:main Dec 3, 2021
12 checks passed

tiran deleted the bootstrap_again branch Dec 3, 2021

bedevere-bot removed the awaiting merge label Dec 3, 2021

tiran mentioned this pull request May 9, 2022

Reintroduce bootstrap_python for freezing #90108

Closed

kumaraditya303 mentioned this pull request Jun 23, 2022

Deepfreeze using 3.10, not bootstrap interpreter. #93771

Closed

corona10 mentioned this pull request Sep 12, 2022

Link-time-optimization with clang is broken #96761

Closed

bpo-45950: Introduce Bootstrap Python again #29859

bpo-45950: Introduce Bootstrap Python again #29859

tiran commented Nov 30, 2021 •

edited

bedevere-bot commented Nov 30, 2021

bedevere-bot commented Nov 30, 2021

gvanrossum commented Nov 30, 2021

vstinner commented Nov 30, 2021

vstinner commented Nov 30, 2021

vstinner commented Nov 30, 2021

tiran commented Dec 1, 2021 •

edited

gvanrossum left a comment

hroncok commented Dec 2, 2021

hroncok commented Dec 2, 2021

tiran commented Dec 2, 2021

tiran commented Dec 2, 2021

bedevere-bot commented Dec 2, 2021

hroncok commented Dec 2, 2021

tiran commented Dec 3, 2021

bedevere-bot commented Dec 3, 2021

bpo-45950: Introduce Bootstrap Python again #29859

bpo-45950: Introduce Bootstrap Python again #29859

Conversation

tiran commented Nov 30, 2021 • edited

bedevere-bot commented Nov 30, 2021

bedevere-bot commented Nov 30, 2021

gvanrossum commented Nov 30, 2021

vstinner commented Nov 30, 2021

vstinner commented Nov 30, 2021

vstinner commented Nov 30, 2021

tiran commented Dec 1, 2021 • edited

gvanrossum left a comment

hroncok commented Dec 2, 2021

hroncok commented Dec 2, 2021

tiran commented Dec 2, 2021

tiran commented Dec 2, 2021

bedevere-bot commented Dec 2, 2021

hroncok commented Dec 2, 2021

tiran commented Dec 3, 2021

bedevere-bot commented Dec 3, 2021

tiran commented Nov 30, 2021 •

edited

tiran commented Dec 1, 2021 •

edited