bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818

koubaa · 2020-08-10T21:11:59Z

https://bugs.python.org/issue1635741

koubaa · 2020-08-12T01:57:44Z

@shihai1991 @vstinner @corona10 FYI. The macOS test failure looks to be due to the following error which seems unrelated to my change:

module 'test.support' has no attribute 'EnvironmentVarGuard'

shihai1991 · 2020-08-12T04:00:21Z

@shihai1991 @vstinner @corona10 FYI. The macOS test failure looks to be due to the following error which seems unrelated to my change:

module 'test.support' has no attribute 'EnvironmentVarGuard'

Hm, this failure have no relation with your PR. It's weird. Some failed test cases have updated: https://github.com/python/cpython/blob/master/Lib/test/test_selectors.py#L540

shihai1991 · 2020-08-12T04:12:01Z

The failed test cases have been fixed in 490c542
Can you do some git rebase operation? MAYBE this operation can solve your probleam.

Modules/_blake2/blake2b_impl.c

vstinner

This PR is a little bit too large. Would you mind to restrict it to blake2 which is already complex enough?

Once this PR is merged, you can write a PR for sha3 which is complex. Then we can finish with remaining modules like md5 and sha1. Or publish all PR are once.

Modules/_blake2/blake2b_impl.c

Modules/_blake2/blake2module.c

bedevere-bot · 2020-08-13T10:30:31Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

koubaa · 2020-08-13T12:17:26Z

@vstinner that's not a problem - I put each file in its own commit so I can easily submit separate PRs. I'll submit one for sha3, one for blake2, and one with the remaining ones. After breaking up and creating PRs for each I'll address your comments

koubaa · 2020-08-13T12:24:20Z

@vstinner

blake 2: bpo-1635741: port blake2 module to multi-phase init #21856
sha3: bpo-1635741: port sha3 module to multi-phase init #21855
everything else: I kept in this PR

Thanks for the review!

Modules/md5module.c

Modules/sha512module.c

koubaa · 2020-08-13T16:28:58Z

@vstinner I'm going on vacation so I'll come back to address you comment next week.

koubaa · 2020-08-22T00:23:21Z

The MacOS/Ubuntu CI has these test failures:

3 tests failed:
test_hashlib test_random test_statistics

Running locally (windows) those tests pass:

./PCBuild/amd64/python.exe -m test test_statistics test_hashlib test_random
0:00:00 Run tests sequentially
0:00:00 [1/3] test_statistics
0:00:08 load avg: 0.00 [2/3] test_hashlib
0:00:09 load avg: 0.00 [3/3] test_random
== Tests result: SUCCESS ==

I rebased on the latest master so I'm not sure what could be causing it.

**** EDIT ****
This was due to a use-after free which is now fixed.

Misc/NEWS.d/next/Core and Builtins/2020-08-10-16-11-32.bpo-1635741.O0d3ym.rst

Modules/md5module.c

Modules/sha1module.c

Modules/md5module.c

Modules/sha512module.c

Modules/md5module.c

Misc/NEWS.d/next/Core and Builtins/2020-08-10-16-11-32.bpo-1635741.O0d3ym.rst

Modules/md5module.c

Modules/sha1module.c

Modules/sha512module.c

Modules/md5module.c

vstinner

LGTM. Thanks for the multiple updates. These PR wasn't easy!

vstinner · 2020-09-04T08:55:28Z

Modules/sha512module.c


-    if (Py_IS_TYPE((PyObject*)self, &SHA512type)) {
-        if ( (newobj = newSHA512object())==NULL)
+    if (Py_IS_TYPE((PyObject*)self, st->sha512_type)) {


Without defining_class, this test was wrong. So thanks for using defining_class :-)

vstinner

LGTM, but please fix the NEWS entry.

Misc/NEWS.d/next/Core and Builtins/2020-08-10-16-11-32.bpo-1635741.O0d3ym.rst

vstinner · 2020-09-06T10:09:59Z

Merged, thanks.

* origin/master: (1373 commits) bpo-1635741: Port mashal module to multi-phase init (python#22149) bpo-1635741: Port _string module to multi-phase init (pythonGH-22148) bpo-1635741: Convert _sha256 types to heap types (pythonGH-22134) bpo-1635741: Port the termios to multi-phase init (PEP 489) (pythonGH-22139) bpo-41732: add iterator to memoryview (pythonGH-22119) bpo-40744: Drop support for SQLite pre 3.7.3 (pythonGH-20909) bpo-41316: Make tarfile follow specs for FNAME (pythonGH-21511) bpo-41720: Add "return NotImplemented" in turtle.Vec2D.__rmul__(). (pythonGH-22092) bpo-1635741 port _curses_panel to multi-phase init (PEP 489) (pythonGH-21986) bpo-1635741: Port _overlapped module to multi-phase init (pythonGH-22051) bpo-1635741: Port _opcode module to multi-phase init (PEP 489) (pythonGH-22050) bpo-1635741 port zlib module to multi-phase init (pythonGH-21995) [doc] Add link to Generic in typing (pythonGH-22125) bpo-41513: Expand comments and add references for a better understanding (pythonGH-22123) bpo-1635741: Port _sha1, _sha512, _md5 to multiphase init (pythonGH-21818) closes bpo-41723: Fix an error in the py_compile documentation. (pythonGH-22110) [doc] Fix padding in some typing definitions (pythonGH-22114) Fix documented Python version for venv --upgrade-deps (pythonGH-22113) bpo-40318: Migrate to SQLite3 trace v2 API (pythonGH-19581) bpo-41687: Fix sendfile implementation to work with Solaris (python#22040) ...

…1818) Port the _sha1, _sha512, and _md5 extension modules to multi-phase initialization API (PEP 489).

koubaa requested a review from tiran as a code owner Aug 10, 2020

the-knights-who-say-ni added the CLA signed label Aug 10, 2020

bedevere-bot added the awaiting review label Aug 10, 2020

koubaa changed the title ~~Bpo 1635741 hashes~~ bpo-1635741: Port hashlib modules to multiphase init (PEP 489) Aug 10, 2020

tiran self-assigned this Aug 11, 2020

vstinner reviewed Aug 12, 2020

View changes

Modules/_blake2/blake2b_impl.c Outdated Show resolved Hide resolved

koubaa force-pushed the bpo-1635741-hashes branch from 28c3f7b to 8491100 Compare Aug 12, 2020

vstinner requested changes Aug 13, 2020

View changes

Modules/_blake2/blake2b_impl.c Outdated Show resolved Hide resolved

Modules/_blake2/blake2module.c Outdated Show resolved Hide resolved

Modules/_blake2/blake2module.c Outdated Show resolved Hide resolved

bedevere-bot added awaiting changes and removed awaiting review labels Aug 13, 2020

koubaa force-pushed the bpo-1635741-hashes branch from 5ddff05 to fddf3ce Compare Aug 13, 2020

vstinner reviewed Aug 13, 2020

View changes

Modules/md5module.c Show resolved Hide resolved

Modules/md5module.c Outdated Show resolved Hide resolved

Modules/md5module.c Show resolved Hide resolved

Modules/sha512module.c Outdated Show resolved Hide resolved

Modules/sha512module.c Show resolved Hide resolved

koubaa force-pushed the bpo-1635741-hashes branch from fddf3ce to 617660d Compare Aug 21, 2020

vstinner reviewed Aug 28, 2020

View changes

vstinner reviewed Aug 29, 2020

View changes

Modules/md5module.c Outdated Show resolved Hide resolved

koubaa added 7 commits Aug 31, 2020

port sha1

4ae1a3f

port sha512

b389566

port md5

8add6b1

blurb

3ff18f7

address comments

9388486

remove unneeded comment

e0c0feb

fix use-after-free

6c7f435

koubaa added 4 commits Aug 31, 2020

add traverse/clear/free module methods

37e2282

resolve comments

0384e6b

pep7

2cfcb12

fix dealloc function

4bcb57f

koubaa force-pushed the bpo-1635741-hashes branch from 02ffcf6 to 4bcb57f Compare Aug 31, 2020

vstinner reviewed Sep 1, 2020

View changes

update based on PR review

3cd5d6e

vstinner reviewed Sep 2, 2020

View changes

Modules/md5module.c Outdated Show resolved Hide resolved

koubaa added 2 commits Sep 2, 2020

use defining_class

0244bcf

remove comment

b66a965

vstinner approved these changes Sep 4, 2020

View changes

bedevere-bot added awaiting merge and removed awaiting changes labels Sep 4, 2020

vstinner approved these changes Sep 4, 2020

View changes

Misc/NEWS.d/next/Core and Builtins/2020-08-10-16-11-32.bpo-1635741.O0d3ym.rst Outdated Show resolved Hide resolved

fix readme

df69769

vstinner merged commit 63f102f into python:master Sep 6, 2020
3 checks passed

bedevere-bot removed the awaiting merge label Sep 6, 2020

koubaa deleted the bpo-1635741-hashes branch Sep 11, 2020

xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020

bpo-1635741: Port _sha1, _sha512, _md5 to multiphase init (pythonGH-2…

5644536

…1818) Port the _sha1, _sha512, and _md5 extension modules to multi-phase initialization API (PEP 489).

kylotan mannequin mentioned this pull request Sep 19, 2022

Py_Finalize() doesn't clear all Python objects at exit #44470

Closed

bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818

bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818

koubaa commented Aug 10, 2020 •

edited by bedevere-bot

koubaa commented Aug 12, 2020 •

edited

shihai1991 commented Aug 12, 2020

shihai1991 commented Aug 12, 2020

vstinner left a comment

bedevere-bot commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 22, 2020 •

edited

vstinner left a comment

vstinner Sep 4, 2020

vstinner left a comment

vstinner commented Sep 6, 2020

bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818

bpo-1635741: Port hashlib modules to multiphase init (PEP 489) #21818

Conversation

koubaa commented Aug 10, 2020 • edited by bedevere-bot

koubaa commented Aug 12, 2020 • edited

shihai1991 commented Aug 12, 2020

shihai1991 commented Aug 12, 2020

vstinner left a comment

bedevere-bot commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 13, 2020

koubaa commented Aug 22, 2020 • edited

vstinner left a comment

vstinner Sep 4, 2020

Choose a reason for hiding this comment

vstinner left a comment

vstinner commented Sep 6, 2020

koubaa commented Aug 10, 2020 •

edited by bedevere-bot

koubaa commented Aug 12, 2020 •

edited

koubaa commented Aug 22, 2020 •

edited