Repository navigation
Conversation
Reuse installed tools, bypass cold nonpublishing runs, keep cache servers inside build containers, and bound pilot main archives independently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Native Linux validation is still needed to verify sccache availability, mounted-cache permissions, container lifecycle hooks, and successful cache population and restored builds with valid launchers and flush statistics.
No open issues 🎉
What changed in this PR
Introduces a scoped sccache pilot for Linux x64 CPU Release CI while preserving existing build outputs and test coverage. It extends the shared cache actions without enabling caching for other Linux variants or Windows workflows.
Changes:
- Adds build-image tool validation, container-local cache lifecycle handling, and generated compiler-launcher checks.
- Uses restored caches on nonpublishing runs, but bypasses sccache on a complete miss; macOS adopts the same selection behavior.
- Adds trusted-main cleanup targeting two Linux pilot snapshots and a 20 GiB soft archive budget, with regression tests and rollout documentation.
Native Linux build validation and warm-cache performance measurements remain pending; no speedup is claimed.
| File | Description |
|---|---|
.github/workflows/reusable_linux_build.yml |
Integrates the opt-in pilot, launcher checks, and cache publication before tests. |
.github/workflows/macos-ci-build-and-test-workflow.yml |
Uses cache-selection output instead of unconditional launcher flags. |
.github/workflows/linux-sccache-maintenance.yml |
Runs regression tests and isolates write permissions to trusted-main cleanup. |
.github/workflows/linux_ci.yml |
Enables the pilot only for x64 CPU Release. |
.github/actions/setup-sccache/test_cache.py |
Covers selection, tool identity, launchers, lifecycle failures, and retention. |
.github/actions/setup-sccache/README.md |
Documents shared behavior, pilot boundaries, retention, and rollout validation. |
.github/actions/setup-sccache/prune_linux_pilot.py |
Prunes scoped main snapshots while preserving the newest baseline. |
.github/actions/setup-sccache/container.sh |
Manages container-local server startup, statistics, shutdown, and flush markers. |
.github/actions/setup-sccache/cache.py |
Implements tool probing, cache identity, selection, launcher verification, and reporting. |
.github/actions/setup-sccache/action.yml |
Adds installed-tool reuse, container inputs, and conditional cache outputs. |
.github/actions/finish-sccache/action.yml |
Handles enabled-cache reporting and distinguishes host from container shutdown. |
Review effort: Balanced
eserscor
marked this pull request as draft
November 11, 2026 16:13
Remove pilot archive maintenance, pruning logic, and retention-only tests. Keep compiler caching and cold nonpublishing-run bypass unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove the pilot opt-in and ccache restore/save branches. Apply sccache to every Linux caller, partition keys by execution providers, and validate CUDA launchers alongside C/C++ while retaining cold nonpublishing-run bypass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
eserscor
changed the title
Install pinned, checksummed x64/arm64 releases in repo-derived images and stage the shared installer before image hashing. Preserve GCC WebGPU plugin PCH correctness by bypassing its launcher, and fix macOS folded-flag command termination. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep GCC's libstdc++ toolchain and retain the GCC PCH safety guard. Cover Clang PCH reuse and header invalidation without changing other Linux compilers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 12, 2026
eserscor
added a commit
that referenced
this pull request
Nov 13, 2026
### Description <!-- Describe your changes. --> Repair portable, rolling Linux ccache snapshots without adopting sccache or changing compilers, PCH, build/security flags, or test coverage. Retain shared snapshots where cold compilation takes at least five minutes; leave shorter builds on their existing local ccache without GitHub snapshot restore/save overhead. - Archive workspace-relative `.cache/ccache` instead of the reusable workflow's `~/.cache/cache` typo or home-relative paths that depend on runner workspace depth. - Share a setup action with a fresh `ccache-linux-v2` namespace. Scope compatibility to build variant/dependency identity, architecture, and actual local Docker image ID or host compiler/ccache versions. Publish unique run-ID/attempt snapshots with only a compatibility-scoped restore prefix; no broad fallback. - Publish snapshots only after successful main builds. PRs can restore but never publish; compatible restores no longer suppress publication of updated entries. Historical archives are not deleted; normal GitHub eviction applies. - Explicitly route the pinned Docker build action to `/onnxruntime_src/.cache/ccache` through its existing environment prefix while preserving caller Python setup. Limit cross-UID permissions and umask changes to compiler-cache contents. - Apply the measured cost cutoff in `655926d690efbc406ba4ce5c14a4fe381315cbb6`: remove both snapshot setup/restore and save from six minimal variants whose **cold build phase was under five minutes**: custom ops, type reduction, globally allowed types, minimal without optional features, extended minimal without optional features, and Android NNAPI. Keep snapshots for exceptions-disabled minimal, extended minimal, regular without optional features, and the unchanged reusable Linux callers. Existing local ccache tools/flags, VCPKG policy, tests, and other settings remain unchanged. The cutoff is a static policy based on cold measurements, not warm timings or a runtime threshold. - Add namespace, publication-policy and eligibility regression tests, portable archive/cross-UID smoke tests, and cache-contract documentation. Declare the tests' PyYAML dependency in `requirements-dev.txt`. **Real GitHub cache-service cold/warm validation — November 13** Ran both existing workflows twice through normal branch-specific push triggers, preserving all six Linux CPU variants, all nine previously snapshot-enabled minimal variants, the full-build/test-data prerequisite, original compiler/PCH/security options and tests. The experiment was isolated on `eserscor-2026/11/13-branch-ccache-experiment`, not added to this PR. | Workflow | Cold run | Warm run | Created-to-last-job completion, cold → warm | | --- | --- | --- | ---: | | Linux CI | [passed](https://github.com/microsoft/onnxruntime/actions/runs/43448889588) | [passed](https://github.com/microsoft/onnxruntime/actions/runs/43458960142) | 59:02 → 49:48 | | Linux CPU Minimal Build E2E | [passed](https://github.com/microsoft/onnxruntime/actions/runs/43448889207) | [passed](https://github.com/microsoft/onnxruntime/actions/runs/43458959669) | 27:17 → 28:47 | **All 32 jobs passed.** All 15 cold compiler-cache restores missed. Every cold snapshot save was confirmed in job logs and the GitHub caches API before the second push. All 15 warm restores matched their corresponding cold publication exactly, with no toolchain/Docker-identity mismatch; all 15 warm saves were also confirmed. This exercised real pinned `actions/cache` restore/save, **not artifact-based compiler-cache transport**. The ordinary `test_data` artifact remained unchanged. Cold commit: `602870c80f0baac6e9c6bd1178dfd91a9f163dc6`. Warm commit: `bf9737d205e44946522a7e7b3d085f4cd98b3bcb`, an intentionally empty commit. Both have identical source tree `571f761d828096a281d5be672f1834ea4c90dd42`. An experiment-only cache prefix kept these snapshots separate from production main keys. No cache deletion, custom maintenance, shared-image modification, `actions:write` grant, manual rerun or GPU workflow expansion was used. Times below are **cold → warm**, in minutes:seconds. Build means the standalone Actions compilation-step wall time; starred native composite/Android rows use `build.py` CMake-build command to test-start/build-complete timestamps within a combined step. Whole job includes setup, cache operations, tests and post steps, but excludes queue/dependency waiting. Hit rates are among cacheable compiler calls, not all tool invocations. Every measured variant restored cold=miss/warm=hit, saved both times, and passed both times. The measurements precede the final six-variant snapshot-policy reduction. | Variant | Build time | Whole-job time | Warm hits / misses | Hit rate | Final shared-snapshot policy | | --- | ---: | ---: | ---: | ---: | --- | | x64 Debug ASan | 17:05 → 2:37 | 41:46 → 27:22 | 1,364 / 1 | 99.93% | Keep | | x64 Release | 16:11 → 0:55 | 33:07 → 16:44 | 1,411 / 2 | 99.86% | Keep | | x64 Release Python 3.14 | 16:03 → 1:03 | 32:18 → 20:05 | 1,411 / 2 | 99.86% | Keep | | x64 MinSizeRel, exceptions disabled | 7:20 → 1:06 | 9:30 → 4:05 | 638 / 1 | 99.84% | Keep | | ARM64 Debug | 17:54 → 3:54 | 57:14 → 43:38 | 1,570 / 1 | 99.94% | Keep | | ARM64 Release | 17:47 → 0:53 | 28:22 → 11:37 | 1,615 / 2 | 99.88% | Keep | | Minimal, exceptions disabled | 5:24 → 1:17 | 8:31 → 5:03 | 622 / 1 | 99.84% | Keep | | Minimal, custom ops* | 4:17.2 → 0:44.6 | 6:46 → 3:27 | 694 / 1 | 99.86% | Omit: cold build <5m | | Minimal, type reduction* | 4:17.0 → 0:34.4 | 7:16 → 3:15 | 686 / 1 | 99.85% | Omit: cold build <5m | | Minimal, globally allowed types* | 4:27.3 → 0:55.0 | 7:14 → 4:18 | 686 / 1 | 99.85% | Omit: cold build <5m | | Extended minimal | 5:58 → 1:36 | 9:17 → 5:01 | 734 / 1 | 99.86% | Keep | | Regular, no optional features | 8:46 → 2:40 | 11:55 → 6:41 | 705 / 2 | 99.72% | Keep | | Minimal, no optional features | 4:33 → 0:40 | 7:12 → 3:53 | 449 / 1 | 99.78% | Omit: cold build <5m | | Extended minimal, no optional features | 4:37 → 0:42 | 6:49 → 4:02 | 470 / 1 | 99.79% | Omit: cold build <5m | | Android NNAPI* | 3:57.3 → 0:13.2 | 6:36 → 3:09 | 1,100 / 1 | 99.91% | Omit: cold build <5m | The unchanged full-ORT prerequisite has no compiler snapshot: its build step took 14:15 → 15:02 and whole job 16:11 → 17:27 ([cold](https://github.com/microsoft/onnxruntime/actions/runs/43448889207/job/131965536019), [warm](https://github.com/microsoft/onnxruntime/actions/runs/43458959669/job/132000462302)). The minimal workflow was **90 seconds slower overall** despite faster snapshot-enabled compilation: that prerequisite was 76 seconds slower, and warm dependent jobs waited 3:53–4:43 for runners versus 8–12 seconds cold. Compiler acceleration is not end-to-end workflow acceleration. This is one sequential pair, not a statistical benchmark; runner, Docker/setup, existing VCPKG-cache and test variability remain. **Why ARM64 Debug remains the long tail** [Warm ARM64 Debug job](https://github.com/microsoft/onnxruntime/actions/runs/43458960142/job/132000464739): compilation fell from **17:54 to 3:54**, but tests remained **35:18 cold / 35:17 warm**, about **81% of the warm job**. Ccache does not accelerate test execution; cache restore/save took roughly ten seconds each. CTest already uses `--parallel 16`, but the **7,019 provider cases execute inside one unsharded `onnxruntime_provider_test` process**. That entry took 2,112.85 seconds cold / 2,112.21 seconds warm; `onnxruntime_test_all` finished in 340.30 seconds warm. Eight provider cases account for 1,417 seconds, approximately 67% of its wall time. - `MatMulNBits` cases total about **14m13s**; `Float16_4b_Accuracy4` alone takes **4m48s** and includes a large 369×6144×2048 regression case. - Two `PackedAttentionTest` random-data cases take **6m40s combined**, despite this being CPU-only. [`RunModelWithRandomInput`](https://github.com/microsoft/onnxruntime/blob/0acfa78a809f0719280f913861a53692252bb8ad/onnxruntime/test/contrib_ops/packed_attention_op_test.cc#L798-L925) generates inputs, weights and biases **before** its `HasCudaEnvironment` check. The two loops make 120 calls and generate approximately 212 million Gaussian weight values even when CUDA is unavailable. The same two tests take about 3.5 seconds combined in the measured ARM64 Release job. - `ReduceSum_batch_by_seq_by_128` takes **3m24s** across 90 large-shape cases; `Gather_overflow_check` takes **1m49s** and deliberately creates an output just over 2 GiB plus reference/output copies. This is reproducible Debug test/fixture cost, not a cache miss or a demonstrated runner stall. Moving CUDA availability checks before unused fixture generation and sharding the provider-test executable are separate potential follow-ups; this cache PR does not skip tests or change Debug optimization/coverage. **Validation** Final eligibility policy at `655926d690efbc406ba4ce5c14a4fe381315cbb6`: - `wsl -d Ubuntu-24.04 --exec bash -c 'source /tmp/ort-ccache-ci-305f4061-venv/bin/activate && python3 -m unittest tools.ci_build.test_linux_ccache -v'` — **7 tests run: 6 passed, 1 optional Docker skip**. - Normalized YAML comparison against the original main baseline — **passed**: both workflows retain identical non-snapshot build/test settings, compiler/PCH/security flags, tool setup, VCPKG policy and coverage. - `actionlint -shellcheck= -pyflakes= .github/workflows/reusable_linux_build.yml .github/workflows/linux_minimal_build.yml` with actionlint 1.7.12, compared with the pre-cutoff baseline — **no new diagnostics**; the same 10 existing custom-runner-label diagnostics remain. Shellcheck and pyflakes disabled for this comparison. - `git diff --check` — **passed**. Final policy committed and pushed; normal PR CI is triggered, not claimed complete here. The four successful experiment runs above validate the repair before the final eligibility-only reduction, not this new head's CI. Earlier repair validation retained as supporting evidence: - `python -m pip install -r requirements-dev.txt` — **passed in clean Linux and Windows virtual environments**, including PyYAML. Both tooling CI jobs and Python format passed after the dependency fix: https://github.com/microsoft/onnxruntime/actions/runs/43354291711 . - `python -m pytest -c .github/workflows/tooling_tests.pytest.ini --rootdir=. -v` — **198 passed, 3 skipped, 206 subtests passed** on the native Linux source copy before the eligibility reduction. - `ORT_CCACHE_DOCKER_IMAGE=ort-ccache-smoke:305f4061 python3 -m unittest tools.ci_build.test_linux_ccache -v` — original **6 tests passed, no skips**, covering differing workspace depths, external-action/direct-Docker environments, UIDs 1001/1002 and host access to container-created entries. - `python -m ruff check tools/ci_build/test_linux_ccache.py` and `python -m ruff format --check tools/ci_build/test_linux_ccache.py` — **passed** on the earlier repair, including repository-pinned ruff 0.12.12 validation. - A prior [artifact-transfer Debug extended-minimal experiment](https://github.com/microsoft/onnxruntime/actions/runs/43443808349) achieved 605.728 seconds cold / 76.928 seconds warm, 1,402 direct hits / zero misses; its temporary workflow/helper were removed. The full-workflow cache-service measurements above supersede that narrower experiment as the principal evidence. Raw logs, exact per-job cache keys, compiler counters, configure/build/test/cache timing JSON and CSV were retained with the experiment report. The main-only publication policy is regression-tested; the experiment validates branch-scoped cache-service publication/consumption, not main-to-PR visibility after merge. No GPU performance claim is made, and unmeasured reusable callers retain their existing policy. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> The reusable workflow saves a path that does not contain its compiler cache, while minimal builds can restore immutable historical archives containing relative traversals to an obsolete runner home/workspace layout. Static exact-match keys also prevent populated snapshots from rolling forward. Fix these root causes while preserving build behavior, and avoid spending shared snapshot resources on short builds that do not determine the workflow's long tail. Independent fix from main, not stacked on draft #34109; that PR and its branch are unchanged. The app branch control only supported a dated kebab-case name with its automatic prefix (`eserscor-2026-11-12-linux-ccache-repair`), not the requested slash-separated branch format. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replace ccache archive integration with sccache for all 14 callers of
reusable_linux_build.yml: CPU configurations/Python variants on x64 and arm64, CUDA/plugin/no-cuDNN, TensorRT, OpenVINO, and WebGPU in-tree/static-plugin builds. No per-variant pilot switch remains. Preserve generators, build/security flags, runner pools, artifacts, and build/test separation, except the explicitly scoped Clang trial below. Windows and workflows outside this reusable workflow do not adopt sccache here.Provisioning and compatibility
/usr/local/bin/sccache, verifies the actual version, and cleans temporary downloads. Unsupported architectures and download/checksum/version failures fail explicitly.build-docker-imageaction andget_docker_image.py, before image hashing. Shared base-image pins/workflows and VM provisioning remain unchanged. Manual Docker builds must also stageinstall_sccache.shinto the context, as documented.Cache behavior and correctness
SCCACHE_DIRECT=false, matching mozilla/sccache#2562. The target override retains the adapter-header include-order invariant; other targets remain cacheable. CPU/test PCH is MSVC-only, and CUDA PCH is Windows-only.build-linux-webgpu-static-plugin-x64-release, allowing that target to retain PCH caching. Install the distribution's Clang in the WebGPU derived image; select Clang explicitly in CMake and the shared probe/update/build/test prefix. Select the existing GCC installation via--gcc-install-dirand explicitly use libstdc++, not libc++; log the selected library. Other jobs retain their existing compilers. Keep the GCC PCH guard for GCC configurations. Regressions cover warm hits, macro-only invalidation, and changed header timestamps/consumer source.build_flagsscalar's trailing newline so appended cache flags do not become a separate shell command.actions: write. Restores work on all refs; new snapshots publish only after successful main compilation without an exact hit.Validation
.\.venv\Scripts\python.exe -m unittest discover -s .github\actions\setup-sccache -p test_cache.py -q— passed: 13 tests, 2 Linux-only compiler regressions skipped on Windows. Covers exact/prefix/miss/main/PR selection, tool identity, container lifecycle/flush/error propagation, launchers, and installer failure/cleanup cases.docker run --rm --mount "type=bind,source=$((Get-Location).Path),target=/repo,readonly" ort-sccache-clang-smoke:c5cb645c bash -c 'set -euo pipefail; python3.11 -m venv --without-pip /tmp/smoke-venv; source /tmp/smoke-venv/bin/activate; cd /repo; python -m unittest discover -s .github/actions/setup-sccache -p test_cache.py -q'— passed: all 13 tests, including real GCC and Clang PCH regressions. Temporary AlmaLinux 8 fixture, nonroot UID 1001, GCC 14.2.1, Clang 21.1.8, sccache 0.19.0. Its first suite attempt failed because the fixture required CMake 3.28 while Alma supplies 3.26.5; reduced only the minimal test fixture's requirement to 3.16, where PCH support was introduced. Production ORT requirements are unchanged.python /smoke/sccache-clang-smoke/abi_smoke.py— passed for GCC-library/Clang-executable and the reverse, exchanging C++ strings/exceptions and checking libstdc++ linkage.python /smoke/sccache-image-smoke/pch_smoke.py— passed cold/warm, header-retouch/new-consumer, and macro-only-header stages. Temporary session-local fixtures, not a full ORT build or blanket ABI guarantee. Equivalent smoke checks also passed on WSL Clang 20.1.8/GCC 13.3.0.python /mnt/c/Users/erscor/.copilot/session-state/c5cb645c-a23b-4ac3-9dca-af95151b53b5/files/sccache-clang-smoke/compare_versions.py— completed in the WSL virtual environment: verified official archive digests and ran isolated GCC PCH reproductions with 0.18.0 and 0.11.0; both produced the expected negative-control failure (stale PCH after the macro change)..\.venv\Scripts\python.exe -m unittest discover -s tools\ci_build -p test_build_args.py -q— passed, 20 tests..\.venv\Scripts\python.exe -m ruff check .github\actions\setup-sccache\test_cache.pyand.\.venv\Scripts\python.exe -m ruff format --check .github\actions\setup-sccache\test_cache.py— passed. Earlier corresponding checks fortools\ci_build\get_docker_image.pyalso passed.actionlint -shellcheck= .github\workflows\linux_webgpu.yml .github\workflows\reusable_linux_build.yml— passed. The separately checked macOS workflow retains only its two established baseline metadata diagnostics.shellcheck tools/ci_build/github/linux/docker/scripts/install_sccache.sh— passed in WSL.git --no-pager diff --check— passed. Structural checks covered all 14 callers/seven installer-enabled Dockerfiles and staging before image hashing.Remaining evidence: Full builds of the actual private CI-derived images, full ORT Clang compilation, native arm64/CUDA compilation, and real ORT cold/warm duration, hit-rate, archive-size, and transfer-time measurements have not been established locally. Normal PR CI must validate actual-image compilation. A cold PR deliberately bypasses caching; main must populate a compatible namespace before subsequent runs can establish warm benefit. No ORT speedup or blanket ABI guarantee is claimed.
Motivation and Context
Expand Linux sccache adoption without a pilot switch or dual cache paths, preserve cold nonpublishing-run bypass and default GitHub cache management, and provision repository-owned image layers without depending on unfinished base-image provisioning. Preserve GCC PCH correctness while trialing Clang for the one affected Linux CI job.