bpo-41909: Enable previously disabled recursion checks. #22536

serhiy-storchaka · 2020-10-04T08:05:51Z

Enable recursion checks which were disabled when get __bases__ of
non-type objects in issubclass() and isinstance() and when intern
strings. It fixes a stack overflow when getting __bases__ leads
to infinite recursion.

Originally recursion checks was disabled for PyDict_GetItem() which
silences all errors including the one raised in case of detected
recursion and can return incorrect result. But now the code uses
PyDict_GetItemWithError() and PyDict_SetDefault() instead.

https://bugs.python.org/issue41909

Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead.

gvanrossum

Can we now get rid of the Py_ALLOW_RECURSION macro? It is undocumented and I find it unlikely that anyone outside the core would be using it (I tried to search GitHub but only found many, many forks of CPython...).

miss-islington · 2020-10-04T21:56:00Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8, 3.9.
🐍🍒⛏🤖

miss-islington · 2020-10-04T21:56:06Z

Sorry @serhiy-storchaka, I had trouble checking out the 3.9 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 9ece9cd65cdeb0a1f6e60475bbd0219161c348ac 3.9

miss-islington · 2020-10-04T21:56:09Z

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 9ece9cd65cdeb0a1f6e60475bbd0219161c348ac 3.8

serhiy-storchaka · 2020-10-04T21:56:23Z

Sure, but in a separate PR.

…GH-22536) Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead. (cherry picked from commit 9ece9cd) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-bot · 2020-10-04T21:57:35Z

GH-22550 is a backport of this pull request to the 3.9 branch.

…GH-22536) Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead.. (cherry picked from commit 9ece9cd) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-bot · 2020-10-04T22:03:40Z

GH-22551 is a backport of this pull request to the 3.8 branch.

…) (GH-22550) Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead. (cherry picked from commit 9ece9cd)

…) (GH-22551) Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead. (cherry picked from commit 9ece9cd)

bedevere-bot · 2020-10-05T00:19:46Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.9 has failed when building commit 7aa22ba.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/429/builds/61) and take a look at the build logs.
Check if the failure is related to this commit (7aa22ba) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/429/builds/61

Failed tests:

test_codecmaps_hk
test_codecmaps_kr

Failed subtests:

test_mapping_file - test.test_codecmaps_kr.TestEUCKRMap
test_mapping_file - test.test_codecmaps_kr.TestJOHABMap
test_mapping_file - test.test_codecmaps_kr.TestCP949Map
test_mapping_file - test.test_codecmaps_hk.TestBig5HKSCSMap

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

403 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 29 sec
test_unparse: 2 min 5 sec
test_peg_generator: 2 min 3 sec
test_multiprocessing_spawn: 2 min 2 sec
test_capi: 2 min 1 sec
test_tokenize: 1 min 34 sec
test_lib2to3: 1 min 25 sec
test_multiprocessing_forkserver: 1 min 21 sec
test_asyncio: 1 min 19 sec
test_unicodedata: 57.5 sec

2 tests failed:
test_codecmaps_hk test_codecmaps_kr

20 tests skipped:
test_devpoll test_gdb test_idle test_ioctl test_kqueue test_msilib
test_ossaudiodev test_smtpnet test_ssl test_startfile test_tcl
test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
test_winconsoleio test_winreg test_winsound test_zipfile64

2 re-run tests:
test_codecmaps_hk test_codecmaps_kr

Total duration: 20 min 51 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/root/buildarea/3.9.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 305, in test_mapping_file
    self._test_mapping_file_plain()
  File "/root/buildarea/3.9.angelico-debian-amd64/build/Lib/test/multibytecodec_support.py", line 319, in _test_mapping_file_plain
    csetval = eval(data[0])
  File "<string>", line 1
    <!DOCTYPE
    ^
SyntaxError: invalid syntax

Enable recursion checks which were disabled when get __bases__ of non-type objects in issubclass() and isinstance() and when intern strings. It fixes a stack overflow when getting __bases__ leads to infinite recursion. Originally recursion checks was disabled for PyDict_GetItem() which silences all errors including the one raised in case of detected recursion and can return incorrect result. But now the code uses PyDict_GetItemWithError() and PyDict_SetDefault() instead.

serhiy-storchaka added type-bug An unexpected behavior, bug, or error needs backport to 3.8 needs backport to 3.9 labels Oct 4, 2020

serhiy-storchaka requested a review from gvanrossum Oct 4, 2020

the-knights-who-say-ni added the CLA signed label Oct 4, 2020

bedevere-bot added the awaiting core review label Oct 4, 2020

serhiy-storchaka force-pushed the enable-recursion-check-in-bases branch from 4e60463 to 5b1438e Compare Oct 4, 2020

gvanrossum approved these changes Oct 4, 2020

View changes

bedevere-bot added awaiting merge and removed awaiting core review labels Oct 4, 2020

serhiy-storchaka merged commit 9ece9cd into python:master Oct 4, 2020
3 checks passed

bedevere-bot removed the awaiting merge label Oct 4, 2020

serhiy-storchaka deleted the enable-recursion-check-in-bases branch Oct 4, 2020

miss-islington assigned serhiy-storchaka Oct 4, 2020

bedevere-bot removed the needs backport to 3.9 label Oct 4, 2020

bedevere-bot removed the needs backport to 3.8 label Oct 4, 2020

serhiy-storchaka removed their assignment Dec 29, 2020

bpo-41909: Enable previously disabled recursion checks. #22536

bpo-41909: Enable previously disabled recursion checks. #22536

serhiy-storchaka commented Oct 4, 2020 •

edited by bedevere-bot

gvanrossum left a comment

miss-islington commented Oct 4, 2020

miss-islington commented Oct 4, 2020

miss-islington commented Oct 4, 2020

serhiy-storchaka commented Oct 4, 2020

bedevere-bot commented Oct 4, 2020

bedevere-bot commented Oct 4, 2020

bedevere-bot commented Oct 5, 2020

bpo-41909: Enable previously disabled recursion checks. #22536

bpo-41909: Enable previously disabled recursion checks. #22536

Conversation

serhiy-storchaka commented Oct 4, 2020 • edited by bedevere-bot

gvanrossum left a comment

miss-islington commented Oct 4, 2020

miss-islington commented Oct 4, 2020

miss-islington commented Oct 4, 2020

serhiy-storchaka commented Oct 4, 2020

bedevere-bot commented Oct 4, 2020

bedevere-bot commented Oct 4, 2020

bedevere-bot commented Oct 5, 2020

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

serhiy-storchaka commented Oct 4, 2020 •

edited by bedevere-bot