Give the small Windows checks one box per image instead of one box per check - #9143
Conversation
The three ubuntu-latest jobs differed only in model, port and test body; checkout, apt, setup-node, setup-python and install.sh --local --no-torch were identical in all three. Windows and macOS already run these three phases in a single job. This is the Linux port of that layout, step bodies unchanged. studio-windows-inference-smoke.yml carried a note saying the trade was not worth making on ubuntu, because setup there is ~1.9 min against ~3.0 min of tests and the sequential wall time would cost more than the slots are worth. The arithmetic was right; the premise that the three jobs start together is not what the runners do. Run 32089506294 started two jobs at 02:40:11 and the third at 02:46:52, then finished at 02:52:57: a 12m46s wall for about 6 min of work. Run 32089558062 staggered all three. Across 18 recent multi-job runs on main the median gap between a run's first and last job start was 175s, and 13 of the 18 exceeded 60s. Bundled is about 7m35 every time. When all three do get slots at once that is ~1m30 slower than the 6m05 longest job; when they do not it is 2-5 min faster. It also returns 2 of the 60 concurrent slots, which is what makes the contended case rarer for every other workflow. Each phase keeps its own model cache directory, HF_HOME, port, server log, artifact and step timeouts, and gates on the shared preamble rather than on the phase before it, so a phase-1 failure still lets phases 2 and 3 report. Phase 3 moves to hf-cache-vision with a key bump because phase 1 keeps hf-cache, whose key is byte-shared with the macOS and Windows gemma phases. tests/studio/test_gguf_smoke_phases_stay_independent.py asserts all of that from the workflow rather than from a list, since every one of these regressions is silent rather than red.
… context Actions rejects the whole workflow before scheduling a job, so the first staging push produced a run with zero jobs and only 'This run likely failed because of a workflow file issue'. Job-level env can read github, needs, strategy, matrix, vars, inputs and secrets, not runner.
Green end to end on a staging repo with an empty actions cache, so every model downloaded cold: 6m03s for all three phases, of which 105s is the now-shared setup. The phases themselves are 26s / 179s / 36s. That is at or under the 6m05 the longest of the three jobs took on its own, on a third of the runners, so the earlier estimate of ~7m35 was pessimistic.
…r check pester, no-vs-gpu-resolve and the two cells each of vs-integration and vcredist-clean-box are six Windows job-runs that execute for 16-34s apiece. On this repo's Windows pool that is not what they cost. Measured over recent main runs, every Windows job waits 2600-3400s for a slot regardless of what it then does: exec queue job 16s 2606s real-VS detection (VS 2022) 22s 3315s real-VS detection (VS 2026) 24s 3170s VC++ round-trip (windows-latest) 26s 3358s VC++ round-trip (windows-2025-vs2026) 33s 3311s setup.ps1 unit tests 34s 3391s GPU prebuilt resolves without VS 1108s 3164s Chat UI Tests So six slots deliver 155s of work, and the queue those slots help create is what the 18-minute Chat UI job waits in. Merged by runner image they are three job-runs and nothing moves platform: windows-2022 keeps its own box for VS 2022 detection, windows-2025-vs2026 keeps its own for VS 2026, and windows-latest carries the three checks that only need a stock Windows box. Ordering inside a box is load-bearing. The VC++ phase uninstalls the runtime and restores the registry in a finally, so it runs last everywhere. The phases before it install a PowerShell module, pip packages and a simulated no-build-tools tree under the workspace; none writes the VC++ registry keys the round-trip reads as ground truth, so its clean-box precondition still holds. The two long jobs (inference-smoke 718s, no-vs-cpu 464s) stay on their own runners: their cost is execution, not slot occupancy. tests/studio/test_windows_small_checks_stay_on_their_image.py pins the phase-to-image contract as data rather than reading it back out of the workflow, because a check running on the wrong Windows image mostly still passes. Stacked on #9139, which edits a comment in this file. Not verifiable locally: needs a cross-platform staging run.
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bd305d43d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return dict(found) | ||
|
|
||
|
|
||
| def test_the_workflow_is_a_single_job(): |
There was a problem hiding this comment.
These new guard tests—and the companion test_windows_small_checks_stay_on_their_image.py module—are never collected by CI. The repository's default pytest configuration limits discovery to tests/security, while workflow-trigger-lint.yml invokes a fixed list of individual Studio guard modules that does not include either new file. Consequently, a later workflow-only change can violate the phase-isolation or image-placement invariants and still pass every check, so add both modules to the workflow-trigger lint job or another unfiltered CI invocation.
Useful? React with 👍 / 👎.
test_pester_bootstrap_hardening.py keyed on jobs['pester'], so grouping the small Windows checks by runner image turned all 8 of its assertions red on a rename that changed nothing they assert. The staged run caught it: 'Guard the Pester bootstrap' failed while every phase after it passed. Looks up whichever job installs Pester instead. Worth noting that the gating held exactly as designed under a real failure: the guard failed and the no-VS resolve and VC++ round-trip phases on the same box still ran and passed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24ccf34115
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - name: Wait for /api/health, log in, change password, load model | ||
| id: ready-vision | ||
| timeout-minutes: 15 | ||
| if: ${{ !cancelled() && steps.sdks.outcome == 'success' && steps.prime-hf-vision.outcome != 'failure' }} |
There was a problem hiding this comment.
If the boot-vision step fails—for example because the server exits before binding—this condition still runs the health/login/load step because it only checks the model-prime outcome. The step then polls a port with no server for up to 180 seconds and produces a second, misleading failure; require steps.boot-vision.outcome == 'success' here, as the API and tool phases already do.
Useful? React with 👍 / 👎.
1 participant
pester, no-vs-gpu-resolve and the two cells each of vs-integration and
vcredist-clean-box are six Windows job-runs that execute for 16-34s apiece. On
this repo's Windows pool that is not what they cost. Measured over recent main
runs, every Windows job waits 2600-3400s for a slot regardless of what it then
does:
exec queue job
16s 2606s real-VS detection (VS 2022)
22s 3315s real-VS detection (VS 2026)
24s 3170s VC++ round-trip (windows-latest)
26s 3358s VC++ round-trip (windows-2025-vs2026)
33s 3311s setup.ps1 unit tests
34s 3391s GPU prebuilt resolves without VS
1108s 3164s Chat UI Tests
So six slots deliver 155s of work, and the queue those slots help create is what
the 18-minute Chat UI job waits in. Merged by runner image they are three
job-runs and nothing moves platform: windows-2022 keeps its own box for VS 2022
detection, windows-2025-vs2026 keeps its own for VS 2026, and windows-latest
carries the three checks that only need a stock Windows box.
Ordering inside a box is load-bearing. The VC++ phase uninstalls the runtime and
restores the registry in a finally, so it runs last everywhere. The phases before
it install a PowerShell module, pip packages and a simulated no-build-tools tree
under the workspace; none writes the VC++ registry keys the round-trip reads as
ground truth, so its clean-box precondition still holds. The two long jobs
(inference-smoke 718s, no-vs-cpu 464s) stay on their own runners: their cost is
execution, not slot occupancy.
tests/studio/test_windows_small_checks_stay_on_their_image.py pins the
phase-to-image contract as data rather than reading it back out of the workflow,
because a check running on the wrong Windows image mostly still passes.
Stacked on #9139, which edits a comment in this file.
Not verifiable locally: needs a cross-platform staging run.