Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-40521: Per-interpreter interned strings #20085

Merged
merged 1 commit into from Dec 26, 2020
Merged

bpo-40521: Per-interpreter interned strings #20085

merged 1 commit into from Dec 26, 2020

Conversation

@vstinner
Copy link
Member

@vstinner vstinner commented May 14, 2020 •
edited by bedevere-bot

Make the Unicode dictionary of interned strings per-interpreter to
make it compatible with subinterpreters.

https://bugs.python.org/issue40521
@vstinner
Copy link
Member Author

@vstinner vstinner commented May 14, 2020

This change requires https://bugs.python.org/issue39465 to be fixed, and also to make the type method cache compatible with subinterpreters.

@vstinner vstinner force-pushed the unicode_interned_subinterp branch from d86790c to 342c3ed May 14, 2020
@vstinner vstinner mentioned this pull request Jun 7, 2020
Merged
@vstinner
Copy link
Member Author

@vstinner vstinner commented Dec 23, 2020

This change requires https://bugs.python.org/issue39465 to be fixed, and also to make the type method cache compatible with subinterpreters.

With this PR, tests using subinterpreters like test_ast.test_subinterpreter() does crash since the method cache is built by default, whereas it shares interned strings between multiple interpreters. MCACHE must not be defined to test this PR, or the method cache must be fixed for subinterpreters.

@vstinner vstinner force-pushed the unicode_interned_subinterp branch from 342c3ed to b11a338 Dec 23, 2020
@vstinner
Copy link
Member Author

@vstinner vstinner commented Dec 23, 2020

I rebased my PR on master. It is still a draft: PR #20058 must be merged first, and then the method cache must be fixed for subinterpreter, before we can consider to merge this PR.

@vstinner vstinner mentioned this pull request Dec 23, 2020
Merged
@vstinner vstinner mentioned this pull request Dec 26, 2020
Merged
@vstinner vstinner force-pushed the unicode_interned_subinterp branch 2 times, most recently from 0bbf371 to 2420a96 Dec 26, 2020
@vstinner vstinner marked this pull request as ready for review Dec 26, 2020
@vstinner vstinner requested a review from markshannon as a code owner Dec 26, 2020
@vstinner vstinner changed the title [WIP] bpo-40521: Per-interpreter interned strings bpo-40521: Per-interpreter interned strings Dec 26, 2020
@vstinner vstinner force-pushed the unicode_interned_subinterp branch from 2420a96 to 5b46b09 Dec 26, 2020
@vstinner
Copy link
Member Author

@vstinner vstinner commented Dec 26, 2020

And I pushed a second fix for test_repl:

_PyUnicode_ClearInterned() now uses PyDict_Next() to no longer allocate memory, to ensure that the interned dictionary is cleared.

It's more an enhancement than a fix. Previously, the code also failed. It's just that previously, we didn't check if interned is NULL at exit, whereas my PR adds such assertion.

@vstinner vstinner force-pushed the unicode_interned_subinterp branch from 5b46b09 to 5aea9de Dec 26, 2020
@vstinner
bpo-40521: Per-interpreter interned strings
8cb4246 
Make the Unicode dictionary of interned strings compatible with
subinterpreters.

Remove the INTERN_NAME_STRINGS macro in typeobject.c: names are
always now interned (even if EXPERIMENTAL_ISOLATED_SUBINTERPRETERS
macro is defined).

_PyUnicode_ClearInterned() now uses PyDict_Next() to no longer
allocate memory, to ensure that the interned dictionary is cleared.
@vstinner vstinner force-pushed the unicode_interned_subinterp branch from 5aea9de to 8cb4246 Dec 26, 2020
@vstinner vstinner merged commit ea25180 into python:master Dec 26, 2020
11 checks passed
@vstinner vstinner deleted the unicode_interned_subinterp branch Dec 26, 2020
adorilson added a commit to adorilson/cpython that referenced this issue Mar 13, 2021
@vstinner @adorilson
bpo-40521: Per-interpreter interned strings (pythonGH-20085)
68dcee2 
Make the Unicode dictionary of interned strings compatible with
subinterpreters.

Remove the INTERN_NAME_STRINGS macro in typeobject.c: names are
always now interned (even if EXPERIMENTAL_ISOLATED_SUBINTERPRETERS
macro is defined).

_PyUnicode_ClearInterned() now uses PyDict_Next() to no longer
allocate memory, to ensure that the interned dictionary is cleared.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
Revert "bpo-40521: Per-interpreter interned strings (pythonGH-20085)"
d832143 
This reverts commit ea25180.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
Revert "bpo-40521: Per-interpreter interned strings (pythonGH-20085)"
229de2d 
This reverts commit ea25180.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
5bb10cc 
…nGH-20085)"

This reverts commit ea25180.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
4d7c4d1 
…nGH-20085)"

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini() but only for
the main interpreter.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
41acd79 
…nGH-20085)"

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini() but only for
the main interpreter.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
76ec67a 
…nGH-20085)"

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini() but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
05d20f3 
…nGH-20085)"

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini() but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 5, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
8ae5d43 
…nGH-20085)"

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini() but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
vstinner added a commit that referenced this issue Jan 6, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (GH-20085
35d6540 
)" (GH-30422)

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.
vstinner added a commit to vstinner/cpython that referenced this issue Jan 6, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
2efdfc5 
…nGH-20085)" (pythonGH-30422)

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.

(cherry picked from commit 35d6540)
@vstinner vstinner mentioned this pull request Jan 6, 2022
Closed
@vstinner vstinner mentioned this pull request Jan 6, 2022
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Jan 6, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
49b0229 
…nGH-20085)" (pythonGH-30422)

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.

(cherry picked from commit 35d6540)
vstinner added a commit to vstinner/cpython that referenced this issue Jan 6, 2022
@vstinner
bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (pytho…
5e9f5d2 
…nGH-20085)" (pythonGH-30422)

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.

Leave PyInterpreterState structure unchanged to keep the ABI backward
compatibility with Python 3.10.0: just rename "interned" to
"unused_interned".

(cherry picked from commit 35d6540)
vstinner added a commit that referenced this issue Jan 6, 2022
@vstinner
[3.10] bpo-46006: Revert "bpo-40521: Per-interpreter interned strings (…
72c260c 
…GH-20085)" (GH-30422) (GH-30425)

This reverts commit ea25180.

Keep "assert(interned == NULL);" in _PyUnicode_Fini(), but only for
the main interpreter.

Keep _PyUnicode_ClearInterned() changes avoiding the creation of a
temporary Python list object.

Leave the PyInterpreterState structure unchanged to keep the ABI
backward compatibility with Python 3.10.0: rename the "interned"
member to "unused_interned".

(cherry picked from commit 35d6540)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@markshannon markshannon

Assignees
No one assigned
Labels
CLA signed
Projects
None yet
Milestone
No milestone
3 participants
@vstinner @the-knights-who-say-ni @bedevere-bot