Skip to content

Honour download(threads=N) concurrency cap - #3019

Merged
ValueRaider merged 1 commit into
ranaroussi:devfrom
louis75aubin-prog:fix/download-thread-cap
Oct 27, 2026
Merged

ValueRaider merged 1 commit into
ranaroussi:devfrom
louis75aubin-prog:fix/download-thread-cap

Conversation

@louis75aubin-prog

Copy link
Copy Markdown
Contributor

Problem

yf.download(..., threads=2) still runs cpu_count() fetches at once.

_download_impl calls multitasking.set_max_threads(threads). But multitasking creates its pool once, at import time, when @multitasking.task decorates _download_one_threaded. The pool is a Semaphore(cpu_count()). set_max_threads() only updates config["MAX_THREADS"] and never resizes the existing pool. This is checked against multitasking 0.0.12. So the threads argument has had no effect on concurrency.

Repro (no network; patch _download_one with a sleep that records peak concurrency), on an 8-core machine:

threads=2     peak concurrency=8
threads=True  peak concurrency=8

#3015 (Tickers.info / Tickers.calendar) hit the same multitasking pool issue. Its _attr_impl uses a per-call ThreadPoolExecutor for that reason, and the PR said the download() side would get its own PR. This is that PR.

Fix

The threaded path now runs on a per-call concurrent.futures.ThreadPoolExecutor(max_workers=threads). This also removes the 10 ms busy-wait loop on ctx.dfs. The progress bar, error/traceback collection and _DownloadCtx are unchanged: _download_one_threaded / _download_one are the same functions, just submitted to the executor. It follows the same pattern as _attr_impl. One difference: the progress bar is still animated from the worker threads, as before, rather than through as_completed, so the progress output doesn't change.

After the fix:

threads=2     peak concurrency=2
threads=True  peak concurrency=8   (unchanged)

Why not multitasking.createPool(threads=N) per call? createPool mutates global state (config["POOL_NAME"], config["MAX_THREADS"]), and @task reads the current pool from that global at call time. Two concurrent download() calls would replace each other's pool, which undoes the reentrancy work in a710ba8. A per-call executor has no shared state.

Default (threads=True) is deliberately unchanged. The code said min(len(tickers), cpu_count() * 2). Because of this bug, the real cap was always cpu_count(). Honouring the * 2 would double the default request concurrency against Yahoo, which could increase rate limiting. So the PR keeps the observed behaviour, min(len(tickers), cpu_count()), and only makes explicit threads=N work. Happy to switch to * 2 if you prefer the original intent.

multitasking is still imported, for cpu_count() only. It could be swapped for os.cpu_count() and the dependency dropped, but I left that out to keep the PR small.

Tests

  • New tests/test_multi.py::test_threads_int_caps_concurrency (no network): 8 tickers with threads=2; asserts peak concurrency ≤ 2 and that all tickers are returned. It fails on dev (8 not less than or equal to 2) and passes with the fix.
  • python -m unittest tests.test_multi tests.test_download_concurrency: 12 tests OK (including the Tickers: add info and calendar, fetched in parallel #3015 tests).
  • ruff check yfinance/ tests/test_multi.py: clean.
  • Manual: yf.download(['AAPL','MSFT','NVDA','ZZZZNOTREAL'], period='5d', threads=2). The progress bar works, the failed ticker is logged as before, and all 4 columns are returned.

Question (not changed in this PR): YfConfig.network.hide_exceptions in _download_one

_download_one saves, sets to False, then restores YfConfig.network.hide_exceptions. Nothing in yfinance reads network.hide_exceptions. Every reader uses YfConfig.debug.hide_exceptions. This came in with 4f38ecf ("Redesign YfConfig"), which replaced history(..., raise_errors=True) with this toggle. history()'s own deprecation message says the replacement for raise_errors is yf.config.debug.hide_exceptions = False. So debug was probably the intent.

I didn't "fix" it to debug because that would change download()'s behaviour:

  • history() would raise inside download() again. Errors would be recorded as repr(e) instead of the _last_error message added in a710ba8, so the "Failed downloads" log text would change.
  • The toggle is a process-wide global flipped from worker threads. Save/restore across concurrent threads can interleave and leave it stuck at False. While any download runs, every other yfinance call in the process would also see hide_exceptions=False and start raising.

The zero-risk option is to delete the dead toggle (4 lines), which changes no behaviour. Would you like that here, or in a separate PR, or should it be left alone?

multitasking creates its pool once, when @multitasking.task decorates
_download_one_threaded at import, sized to cpu_count(). set_max_threads()
only updates config["MAX_THREADS"] and never resizes that pool, so
threads=N was ignored and every download ran cpu_count() fetches at once.

Run the threaded path on a per-call ThreadPoolExecutor(max_workers=N)
instead, which also replaces the busy-wait loop. threads=True keeps its
effective default of min(len(tickers), cpu_count()).
@ValueRaider
ValueRaider merged commit 19d654c into ranaroussi:dev Oct 27, 2026
2 checks passed
@ValueRaider ValueRaider mentioned this pull request Oct 27, 2026
ValueRaider pushed a commit that referenced this pull request Oct 31, 2026
Since #3019, download() runs its threads on a per-call
ThreadPoolExecutor, so multitasking was only used for cpu_count().
Use os.cpu_count() instead and remove the dependency from
pyproject.toml and meta.yaml.

os.cpu_count() returns None when the count is unknown, where
multiprocessing.cpu_count() raised, so fall back to one thread.
The default thread count is otherwise unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants