Repository navigation
Honour download(threads=N) concurrency cap - #3019
Merged
ValueRaider merged 1 commit intoOct 27, 2026
Merged
Conversation
multitasking creates its pool once, when @multitasking.task decorates _download_one_threaded at import, sized to cpu_count(). set_max_threads() only updates config["MAX_THREADS"] and never resizes that pool, so threads=N was ignored and every download ran cpu_count() fetches at once. Run the threaded path on a per-call ThreadPoolExecutor(max_workers=N) instead, which also replaces the busy-wait loop. threads=True keeps its effective default of min(len(tickers), cpu_count()).
Merged
This was referenced Oct 29, 2026
ValueRaider
pushed a commit
that referenced
this pull request
Oct 31, 2026
Since #3019, download() runs its threads on a per-call ThreadPoolExecutor, so multitasking was only used for cpu_count(). Use os.cpu_count() instead and remove the dependency from pyproject.toml and meta.yaml. os.cpu_count() returns None when the count is unknown, where multiprocessing.cpu_count() raised, so fall back to one thread. The default thread count is otherwise unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
yf.download(..., threads=2)still runscpu_count()fetches at once._download_implcallsmultitasking.set_max_threads(threads). But multitasking creates its pool once, at import time, when@multitasking.taskdecorates_download_one_threaded. The pool is aSemaphore(cpu_count()).set_max_threads()only updatesconfig["MAX_THREADS"]and never resizes the existing pool. This is checked against multitasking 0.0.12. So thethreadsargument has had no effect on concurrency.Repro (no network; patch
_download_onewith a sleep that records peak concurrency), on an 8-core machine:#3015 (
Tickers.info/Tickers.calendar) hit the same multitasking pool issue. Its_attr_impluses a per-callThreadPoolExecutorfor that reason, and the PR said thedownload()side would get its own PR. This is that PR.Fix
The threaded path now runs on a per-call
concurrent.futures.ThreadPoolExecutor(max_workers=threads). This also removes the 10 ms busy-wait loop onctx.dfs. The progress bar, error/traceback collection and_DownloadCtxare unchanged:_download_one_threaded/_download_oneare the same functions, just submitted to the executor. It follows the same pattern as_attr_impl. One difference: the progress bar is still animated from the worker threads, as before, rather than throughas_completed, so the progress output doesn't change.After the fix:
Why not
multitasking.createPool(threads=N)per call?createPoolmutates global state (config["POOL_NAME"],config["MAX_THREADS"]), and@taskreads the current pool from that global at call time. Two concurrentdownload()calls would replace each other's pool, which undoes the reentrancy work in a710ba8. A per-call executor has no shared state.Default (
threads=True) is deliberately unchanged. The code saidmin(len(tickers), cpu_count() * 2). Because of this bug, the real cap was alwayscpu_count(). Honouring the* 2would double the default request concurrency against Yahoo, which could increase rate limiting. So the PR keeps the observed behaviour,min(len(tickers), cpu_count()), and only makes explicitthreads=Nwork. Happy to switch to* 2if you prefer the original intent.multitaskingis still imported, forcpu_count()only. It could be swapped foros.cpu_count()and the dependency dropped, but I left that out to keep the PR small.Tests
tests/test_multi.py::test_threads_int_caps_concurrency(no network): 8 tickers withthreads=2; asserts peak concurrency ≤ 2 and that all tickers are returned. It fails ondev(8 not less than or equal to 2) and passes with the fix.python -m unittest tests.test_multi tests.test_download_concurrency: 12 tests OK (including the Tickers: add info and calendar, fetched in parallel #3015 tests).ruff check yfinance/ tests/test_multi.py: clean.yf.download(['AAPL','MSFT','NVDA','ZZZZNOTREAL'], period='5d', threads=2). The progress bar works, the failed ticker is logged as before, and all 4 columns are returned.Question (not changed in this PR):
YfConfig.network.hide_exceptionsin_download_one_download_onesaves, sets toFalse, then restoresYfConfig.network.hide_exceptions. Nothing in yfinance readsnetwork.hide_exceptions. Every reader usesYfConfig.debug.hide_exceptions. This came in with 4f38ecf ("Redesign YfConfig"), which replacedhistory(..., raise_errors=True)with this toggle.history()'s own deprecation message says the replacement forraise_errorsisyf.config.debug.hide_exceptions = False. Sodebugwas probably the intent.I didn't "fix" it to
debugbecause that would changedownload()'s behaviour:history()would raise insidedownload()again. Errors would be recorded asrepr(e)instead of the_last_errormessage added in a710ba8, so the "Failed downloads" log text would change.False. While any download runs, every other yfinance call in the process would also seehide_exceptions=Falseand start raising.The zero-risk option is to delete the dead toggle (4 lines), which changes no behaviour. Would you like that here, or in a separate PR, or should it be left alone?