Task was destroyed but it is pending! timeout

[michaelzus]

Describe the bug

In case of setting a ClientTimout, we are getting the following issue:

Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<TCPConnector._resolve_host() running at /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages/aiohttp/connector.py:874> wait_for=<Future pending cb=[_chain_future.._call_check_cancel() at /labhome/python_3.10.4/lib/python3.10/asyncio/futures.py:384, Task.task_wakeup()]> cb=[TCPConnector._create_direct_connection..drop_exception() at /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages/aiohttp/connector.py:1155]>

To Reproduce

  client_timeout = ClientTimeout(connect=timeout)
  async with Session(headers=self.headers, timeout=client_timeout) as session:
      async with session.post(self.login_url, **params) as response:
          return response.cookies

Expected behavior

not to see the issue

Logs/tracebacks

Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<TCPConnector._resolve_host() running at /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages/aiohttp/connector.py:874> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /labhome/python_3.10.4/lib/python3.10/asyncio/futures.py:384, Task.task_wakeup()]> cb=[TCPConnector._create_direct_connection.<locals>.drop_exception() at /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages/aiohttp/connector.py:1155]>

Python Version

$ python --version
Python 3.10.4

aiohttp Version

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.8.3
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: 
Author-email: 
License: Apache 2
Location: /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by:

multidict Version

$ python -m pip show multidict
Name: multidict
Version: 6.0.2
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages
Requires: 
Required-by: aiohttp, yarl

yarl Version

$ python -m pip show yarl
Name: yarl
Version: 1.8.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /my_env/git/nlastic-connections/venv/lib/python3.10/site-packages
Requires: idna, multidict
Required-by: aiohttp

OS

CentOS Linux release 7.9.2009 (Core)

Related component

Client

Additional context

problem was solved by the following refactor:
adding "host_resolved.cancel()" to aiohttp/connector.py at line 1160

code block:
`

except asyncio.CancelledError:

        def drop_exception(fut: "asyncio.Future[List[Dict[str, Any]]]") -> None:
            with suppress(Exception, asyncio.CancelledError):
                fut.result()

        host_resolved.add_done_callback(drop_exception)
        host_resolved.cancel()
        raise

`

Code of Conduct

• I agree to follow the aio-libs Code of Conduct

[cnicodeme]

Also having a lot of Sentry errors because of this. Could you please ping us once this has been added to a new version.

[cnicodeme]

It would be nice if we could have a patch that we can implement directly in Python while we wait for a proper fix?

[cnicodeme]

Any news on this ?

[Dreamsorcerer]

Could you write a test that reproduces the problem (and submit that as a PR)?

[cnicodeme]

@Dreamsorcerer Sorry it took so long!

I suspect the real issue is some DNS lookup failure occurring at random times. In order to simulate this, I "patched" the TCPConnector._resolve_host function to sleep in order to simulate that issue.

This yields something close to the same error, which leads me to think I'm on the right path.

The solution suggested by @michaelzus does indeed resolve the issue.

Here's a test that reproduces the problem:

from aiohttp import ClientError, ClientSession, ClientTimeout, TCPConnector
from asyncio import TimeoutError
import asyncio

# Patch to simulate a DNS lookup failure
# I believe this is what causing the random errors
async def delay(self, host, port, traces):
    await asyncio.sleep(20)

TCPConnector._resolve_host_with_throttle = delay


async def main(url):
    async with ClientSession() as session:
        try:
            response = await session.get(url, timeout=ClientTimeout(total=1))
            response.close()
        except (ClientError, TimeoutError):
            # All exceptions thrown by aiohttp is handled here
            print('Timeout error handled, but not pending task')
            return False

        return True

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main('http://httpstat.us/200?sleep=5000'))
    asyncio.get_event_loop().close()

Running it as-is yields the following output in the console:

Timeout error handled, but not pending task
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<delay() done, defined at /home/cx42/httperr.py:7> wait_for= cb=[TCPConnector._create_direct_connection..drop_exception() at /home/cx42/.local/lib/python3.10/site-packages/aiohttp/connector.py:1155]>

Applying the modification from @michaelzus returns the following:

Timeout error handled, but not pending task

(The expected result ; Only the print statement is displayed, no exception is raised)

Tested with aiohttp==3.8.4 and aiohttp==3.8.5

Note that the exception says "coro=<delay() done...". The original exception is "coro=<TCPConnector._resolve_host() running at". (running at vs done). I suspect this is because of the patch and can be disregarded.

Having the suggestion from @michaelzus resolving it makes me believe it.

And finally, here's a full stacktrace via Sentry:
https://cyril-nicodeme.sentry.io/share/issue/7a778cd34f0c467082376b1fbce5f7aa/

I'll push a PR.

[cnicodeme]

see #7608

[Dreamsorcerer]

OK, I see what's happening. The reference to the task gets garbage collected at the end of the function, so if no other connector has a reference to the task, then the task ends up getting dropped.

I think we need to add a set to TCPConnector to track the tasks and then cancel and await them in .close().