wait_for(to_thread)) does not work as expected. Extra documentation or fix needed.

[sync]

BPO	43019
Nosy	@asvetlov, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-25.05:32:51.086>
labels = ['type-bug', '3.9', 'expert-asyncio']
title = 'wait_for(to_thread)) does not work as expected. Extra documentation or fix needed.'
updated_at = <Date 2022-03-17.15:09:05.569>
user = 'https://bugs.python.org/sync'

bugs.python.org fields:

activity = <Date 2022-03-17.15:09:05.569>
actor = 'asvetlov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['asyncio']
creation = <Date 2021-01-25.05:32:51.086>
creator = 'sync'
dependencies = []
files = []
hgrepos = []
issue_num = 43019
keywords = []
message_count = 2.0
messages = ['385602', '415415']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'sync']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43019'
versions = ['Python 3.9']

[sync]

Consider the following:

import asyncio
import time

async def async_test():
    while True:
        await asyncio.sleep(1)
        print("in here - async")

def sync_test():
    while True:
        time.sleep(1)
        print("in here - sync")

async def main():
    async def tryexcept(func):
        try:
            await func
        except asyncio.exceptions.TimeoutError:
            print("error thrown")

    await tryexcept(asyncio.wait_for(async_test(), timeout=5))
    await tryexcept(asyncio.wait_for(asyncio.to_thread(sync_test), timeout=5))
    print("back in the main thread")

asyncio.run(main())

The above will lead to:

in here - async
error thrown
in here - sync
error thrown
back in the main thread
in here - sync
in here - sync
in here - sync
[... continues on forever ...]

It seems that the new thread created by to_thread is never cancelled and continues running forever despite the timeout error thrown. This might lead to some unwarranted (and hidden) behavior depending on the application.

Frankly, I'm unsure if a possible bug fix is even viable as from my knowledge cancelling threads in Python is not recommended. However, if not, I think a documentation update would be helpful.

On my end messing with an existing sync library + to_thread + wait_for led me to believe that the execution had been cancelled, when instead it kept on shooting unexpected web requests.

Thank you.

[asvetlov]

Threads are not interruptable in Python.
The same for concurrent.future executor.
asyncio and wait_for is not special.

I'm not sure should we document it and *if* yes -- where should we do it?

Adding a note to every function could be cumbersome.

[kumaraditya303]

This is just the way threads in Python work, also this is not limited to threads, any blocking call can cause this to happen.

[gvanrossum]

Threads and asyncio are an unholy mix and just lead to endless tears. Maybe we need stronger warnings against mixing them, but users will just ignore that.