Skip to content

ci(test-api): shard pytest 3 ways + cache resolved .venv - #1856

Merged
nicoloboschi merged 1 commit into
mainfrom
ci/shard-test-api
May 29, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
ci/shard-test-api

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

On core changes the workflow's critical path is test-api at ~22 min, with pytest -m "not hs_llm_mat and not hs_llm_core" alone taking 18:48 even with -n 8 --dist loadgroup. Splitting that step three ways via pytest-split brings each shard down to ~7-8 min, and a venv cache keeps the resolved env (~3 min on cold) free across shards and re-runs.

After this lands, the critical path on a full core run drops to whichever job is next — test-python-client-oracle at ~15 min — so the follow-up work is applying the same venv-cache pattern to the other ~9 jobs that run uv sync --all-extras.

What changed

  • test-api becomes a 3-way matrix on shard: [1, 2, 3], named test-api (1/3) … (3/3).
  • hindsight-api-slim/.venv is cached via actions/cache@v5, keyed on uv.lock + hindsight-api-slim/pyproject.toml + .python-version. uv sync --frozen still runs after restore — it's a fast link check when the venv matches the lock.
  • pytest-split is supplied ad-hoc via uv run --with pytest-split so the workspace uv.lock is untouched. --splits/--group filter at collection (before xdist takes over), so they compose with the -n 8 --dist loadgroup baked into addopts.

Expected wall-clock impact

Before After (cold cache) After (warm cache)
test-api (critical path) 22:07 ~10 min (shard 1 cold-syncs venv) ~7-8 min (all shards hit cache)

Critical path of the full workflow drops from 22 min → ~15 min (now bounded by test-python-client-oracle).

Out of scope (follow-up)

Apply the same .venv cache pattern to the other jobs that run uv sync --all-extras: test-python-client-oracle, test-doc-examples (×4), test-rust-cli, test-typescript-client*, test-integration, Core LLM tests. Each save is ~2-3 min of CI minutes, but the wall-time win only matters once test-api stops being the critical path — which is what this PR does. Splitting it out keeps the cache-key surface easy to reason about if anything misbehaves.

Test plan

  • First CI run on this PR exercises a cold cache — shard 1 runs uv sync (~3 min), shards 2/3 should hit the cache once shard 1 finishes (or also run cold if they start concurrently — that's fine, only shard 1's save matters)
  • A second CI run (push a no-op commit) verifies all three shards hit the cache (~5-10s on the venv-restore step)
  • Each shard reports ~⅓ of the test count in pytest's summary line, confirming --splits 3 --group N partitions correctly
  • No test is run twice across shards (pytest-split partitions by node-id hash, so partitions are disjoint)
  • All three shards pass green

test-api was the critical-path job at ~22 min on core changes: the
`pytest -m "not hs_llm_mat and not hs_llm_core"` step alone took 18:48
even with `-n 8 --dist loadgroup`. Splitting it across 3 jobs via
pytest-split brings each shard down to ~7-8 min and drops the workflow
critical path to whichever job is next (test-python-client-oracle at
~15 min).

The shards run identical setup, so without a venv cache we'd triple the
~3-min `uv sync --all-extras` cost. Adding actions/cache@v5 on
hindsight-api-slim/.venv keyed on uv.lock + the API pyproject + the
pinned Python version lets shards 2+ skip the expensive resolve/link
on the first run after a lock change, and all three shards hit on
re-runs. `uv sync --frozen` still runs after restore — it's a fast link
check when the venv matches.

pytest-split is added via `uv run --with pytest-split` so the managed
uv.lock stays untouched; --splits/--group filter at collection, before
xdist takes over, so they compose with the existing addopts.

Out of scope: applying the same venv-cache pattern to the other ~9 jobs
that also run `uv sync --all-extras` (test-python-client-oracle,
test-doc-examples (×4), test-rust-cli, test-typescript-client*,
test-integration, Core LLM tests). That's a follow-up — each adds risk
of cache-key drift and the savings only matter once test-api stops
being the critical path.
@nicoloboschi
nicoloboschi merged commit 86b8023 into main May 29, 2026
76 of 77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant