bpo-33997: Fix racing condition in termination of pool result_handler #8009
Conversation
Fixes a racing condition when using Pool.terminate(). Sporadically the poll() function signals new data in the queue but get() never returns. This fix just consumes the two issued sentinels by the cleanup process.
|
Hello, and thanks for your contribution! I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA). Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue. When your account is ready, please add a comment in this pull request Thanks again for your contribution, we look forward to reviewing it! |
|
Added missing "GitHub Name" entry to successful verify CLA. |
| @@ -497,6 +499,7 @@ def _handle_results(outqueue, get, cache): | |||
|
|
|||
| if task is None: | |||
| util.debug('result handler ignoring extra sentinel') | |||
| sentinel_counter += 1 | |||
There was a problem hiding this comment.
This extra sentinel is no longer ignored, so the debug is wrong.
Fixed debug message.
|
This PR is stale because it has been open for 30 days with no activity. |
|
The following commit authors need to sign the Contributor License Agreement: |
|
This PR is stale because it has been open for 30 days with no activity. |
This patch fixes a racing condition when using Pool.terminate()
My application is using imap() but aborting further processing after some iterations. Therefore I used terminate() to end the workers pool but the called join() of the result_handler was hanging sporadically.
I traced the issue down to the problem that somehow poll() signals new data in the outqueue but the preceding get() never returned due to ERROR_IO_PENDING in winapi of the queue hand endless hanging WaitForMultipleObjects().
Waiting at most for the two sentinels issued by the pool cleanup fixes the issue. Also the
while sentinel_count < 2seems to me more reasonable and plattform-independent than the oldfor i in range(10).https://bugs.python.org/issue33997