Repository navigation
ci(test-api): shard pytest 3 ways + cache resolved .venv - #1856
Merged
Merged
Conversation
test-api was the critical-path job at ~22 min on core changes: the `pytest -m "not hs_llm_mat and not hs_llm_core"` step alone took 18:48 even with `-n 8 --dist loadgroup`. Splitting it across 3 jobs via pytest-split brings each shard down to ~7-8 min and drops the workflow critical path to whichever job is next (test-python-client-oracle at ~15 min). The shards run identical setup, so without a venv cache we'd triple the ~3-min `uv sync --all-extras` cost. Adding actions/cache@v5 on hindsight-api-slim/.venv keyed on uv.lock + the API pyproject + the pinned Python version lets shards 2+ skip the expensive resolve/link on the first run after a lock change, and all three shards hit on re-runs. `uv sync --frozen` still runs after restore — it's a fast link check when the venv matches. pytest-split is added via `uv run --with pytest-split` so the managed uv.lock stays untouched; --splits/--group filter at collection, before xdist takes over, so they compose with the existing addopts. Out of scope: applying the same venv-cache pattern to the other ~9 jobs that also run `uv sync --all-extras` (test-python-client-oracle, test-doc-examples (×4), test-rust-cli, test-typescript-client*, test-integration, Core LLM tests). That's a follow-up — each adds risk of cache-key drift and the savings only matter once test-api stops being the critical path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On core changes the workflow's critical path is test-api at ~22 min, with
pytest -m "not hs_llm_mat and not hs_llm_core"alone taking 18:48 even with-n 8 --dist loadgroup. Splitting that step three ways via pytest-split brings each shard down to ~7-8 min, and a venv cache keeps the resolved env (~3 min on cold) free across shards and re-runs.After this lands, the critical path on a full core run drops to whichever job is next —
test-python-client-oracleat ~15 min — so the follow-up work is applying the same venv-cache pattern to the other ~9 jobs that runuv sync --all-extras.What changed
test-apibecomes a 3-way matrix onshard: [1, 2, 3], namedtest-api (1/3)…(3/3).hindsight-api-slim/.venvis cached viaactions/cache@v5, keyed onuv.lock + hindsight-api-slim/pyproject.toml + .python-version.uv sync --frozenstill runs after restore — it's a fast link check when the venv matches the lock.pytest-splitis supplied ad-hoc viauv run --with pytest-splitso the workspaceuv.lockis untouched.--splits/--groupfilter at collection (before xdist takes over), so they compose with the-n 8 --dist loadgroupbaked intoaddopts.Expected wall-clock impact
Critical path of the full workflow drops from 22 min → ~15 min (now bounded by
test-python-client-oracle).Out of scope (follow-up)
Apply the same
.venvcache pattern to the other jobs that runuv sync --all-extras:test-python-client-oracle,test-doc-examples (×4),test-rust-cli,test-typescript-client*,test-integration,Core LLM tests. Each save is ~2-3 min of CI minutes, but the wall-time win only matters once test-api stops being the critical path — which is what this PR does. Splitting it out keeps the cache-key surface easy to reason about if anything misbehaves.Test plan
uv sync(~3 min), shards 2/3 should hit the cache once shard 1 finishes (or also run cold if they start concurrently — that's fine, only shard 1's save matters)--splits 3 --group Npartitions correctly