gh-86128: Add warning to ThreadPoolExecutor docs #94008
Open
+8
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
See python#86128 (comment) for an example where this happens. Essentially, even if it looks like you added an `atexit` handler to instruct your thread to exit gracefully, it will only be executed _after_ your thread has finished. For long-running threads (e.g. threads listening to a queue), that may never happen at all. Elsewhere in python#86128, it's recommended that `ThreadPoolExecutor` not be used for long-running tasks, but this was not reflected in the documentation. Based solely on the API, there is no reason to think you shouldn't use it for long-running tasks. The only reason appears to be a limitation in its implementation, so that should be made explicit in the docs.
4304fc6
to
2bde132
Compare
|
(Note that while this is related to #86128, it does not fix that issue, merely documenting the behavior.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
ThreadPoolExecutor has unusual atexit behavior that can cause threads to block indefinitely. This PR documents that limitation, and adds an explicit warning to the documentation for ThreadPoolExecutor.
See #86128 (comment) for an example where this happens. Essentially, even if it looks like you added an
atexithandler to instruct your thread to exit gracefully, it will only be executed after your thread has finished. For long-running threads (e.g. threads listening to a queue), that may never happen at all.Elsewhere in #86128, it's recommended that
ThreadPoolExecutornot be used for long-running tasks, but this was not reflected in the documentation. Based solely on the API, there is no reason to think you shouldn't use it for long-running tasks. The only reason appears to be a limitation in its implementation, so that should be made explicit in the docs.Thanks very much reviewing!
I'm also open to attempting to fix the underlying odd behavior, though I don't think I have enough context to know what the right fix would be.