[staging CI] unslothai/unsloth#9143 - #1036
Closed
danielhanchen wants to merge 6 commits into
Closed
Conversation
The three ubuntu-latest jobs differed only in model, port and test body; checkout, apt, setup-node, setup-python and install.sh --local --no-torch were identical in all three. Windows and macOS already run these three phases in a single job. This is the Linux port of that layout, step bodies unchanged. studio-windows-inference-smoke.yml carried a note saying the trade was not worth making on ubuntu, because setup there is ~1.9 min against ~3.0 min of tests and the sequential wall time would cost more than the slots are worth. The arithmetic was right; the premise that the three jobs start together is not what the runners do. Run 32089506294 started two jobs at 02:40:11 and the third at 02:46:52, then finished at 02:52:57: a 12m46s wall for about 6 min of work. Run 32089558062 staggered all three. Across 18 recent multi-job runs on main the median gap between a run's first and last job start was 175s, and 13 of the 18 exceeded 60s. Bundled is about 7m35 every time. When all three do get slots at once that is ~1m30 slower than the 6m05 longest job; when they do not it is 2-5 min faster. It also returns 2 of the 60 concurrent slots, which is what makes the contended case rarer for every other workflow. Each phase keeps its own model cache directory, HF_HOME, port, server log, artifact and step timeouts, and gates on the shared preamble rather than on the phase before it, so a phase-1 failure still lets phases 2 and 3 report. Phase 3 moves to hf-cache-vision with a key bump because phase 1 keeps hf-cache, whose key is byte-shared with the macOS and Windows gemma phases. tests/studio/test_gguf_smoke_phases_stay_independent.py asserts all of that from the workflow rather than from a list, since every one of these regressions is silent rather than red.
… context Actions rejects the whole workflow before scheduling a job, so the first staging push produced a run with zero jobs and only 'This run likely failed because of a workflow file issue'. Job-level env can read github, needs, strategy, matrix, vars, inputs and secrets, not runner.
Green end to end on a staging repo with an empty actions cache, so every model downloaded cold: 6m03s for all three phases, of which 105s is the now-shared setup. The phases themselves are 26s / 179s / 36s. That is at or under the 6m05 the longest of the three jobs took on its own, on a third of the runners, so the earlier estimate of ~7m35 was pessimistic.
…r check pester, no-vs-gpu-resolve and the two cells each of vs-integration and vcredist-clean-box are six Windows job-runs that execute for 16-34s apiece. On this repo's Windows pool that is not what they cost. Measured over recent main runs, every Windows job waits 2600-3400s for a slot regardless of what it then does: exec queue job 16s 2606s real-VS detection (VS 2022) 22s 3315s real-VS detection (VS 2026) 24s 3170s VC++ round-trip (windows-latest) 26s 3358s VC++ round-trip (windows-2025-vs2026) 33s 3311s setup.ps1 unit tests 34s 3391s GPU prebuilt resolves without VS 1108s 3164s Chat UI Tests So six slots deliver 155s of work, and the queue those slots help create is what the 18-minute Chat UI job waits in. Merged by runner image they are three job-runs and nothing moves platform: windows-2022 keeps its own box for VS 2022 detection, windows-2025-vs2026 keeps its own for VS 2026, and windows-latest carries the three checks that only need a stock Windows box. Ordering inside a box is load-bearing. The VC++ phase uninstalls the runtime and restores the registry in a finally, so it runs last everywhere. The phases before it install a PowerShell module, pip packages and a simulated no-build-tools tree under the workspace; none writes the VC++ registry keys the round-trip reads as ground truth, so its clean-box precondition still holds. The two long jobs (inference-smoke 718s, no-vs-cpu 464s) stay on their own runners: their cost is execution, not slot occupancy. tests/studio/test_windows_small_checks_stay_on_their_image.py pins the phase-to-image contract as data rather than reading it back out of the workflow, because a check running on the wrong Windows image mostly still passes. Stacked on unslothai#9139, which edits a comment in this file. Not verifiable locally: needs a cross-platform staging run.
for more information, see https://pre-commit.ci
test_pester_bootstrap_hardening.py keyed on jobs['pester'], so grouping the small Windows checks by runner image turned all 8 of its assertions red on a rename that changed nothing they assert. The staged run caught it: 'Guard the Pester bootstrap' failed while every phase after it passed. Looks up whichever job installs Pester instead. Worth noting that the gating held exactly as designed under a real failure: the guard failed and the no-VS resolve and VC++ round-trip phases on the same box still ran and passed.
Owner
Author
|
Staging run finished; closing. Staging PRs exist to run CI on a spare queue and are never merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disposable CI run for unslothai#9143. Do not merge; closed after CI.