Repository navigation
[v3-4-test] Accelerate CI image materialization with same-run snapshots (#74173) - #76183
Merged
Merged
Conversation
…ts (#74173) * Restore the CI image from a snapshot of Docker's data directory Every job that prepares the CI image runs `docker image load` on the stash, which unpacks and checksums every layer of an 8 GB image again, about two minutes per job. Extracting a copy of the image store into a stopped daemon gives the same image in about half a minute on the same runners. The image stash stays as it is, for other consumers and as the fallback when a snapshot does not fit the daemon a job runs on. * Recover Docker safely when CI snapshot materialization fails Snapshot failures must not prevent the authoritative stash from loading, and an older branch snapshot must not substitute an image for the current checkout. * Scope CI snapshots to the current workflow run Raw daemon snapshots are job handoffs, so publishing them into branch caches unnecessarily exposes later runs to arbitrary checkout refs. * Publish authoritative caches before optional CI snapshot execution The snapshot script runs checkout code and prunes Docker state. Existing cache publications must finish before that optional execution. * Isolate CI snapshot creation from branch cache publishers The optional snapshot producer executes checkout code without publishing branch caches; its artifacts are available only to consumers in the same workflow run. * Bind optional snapshot jobs to the workflow commit Snapshot producers must not accept arbitrary checkout refs in privileged workflow contexts, and a partially failed Docker stop must still trigger restart cleanup. * Avoid reloading the CI image before snapshot creation Preparing the authoritative image in a second runner adds nearly three minutes before consumers can start. All branch publications must finish before the optional snapshot code prunes Docker state. * Execute the snapshot script from the immutable workflow commit The builder can check out a configurable ref before publishing caches. Snapshot execution must read the workflow commit itself rather than trust files left in that working tree. * Keep optional snapshots in an isolated read-only job Executing snapshot code inside the configurable-ref builder introduces a cache-poisoning finding. A separate job preserves the security boundary; its latency cost must remain explicit when measuring the runner-time benefit. * Preserve Docker state when snapshot inventory checks fail An unavailable daemon must not be mistaken for an empty image store before destructive snapshot materialization. * Retrigger CI after provider compatibility timeout * Create CI snapshots in the builder after publishing caches Reloading the freshly built image in a second job delays every consumer by nearly four minutes. Trusted workflow commands reuse the builder daemon without executing code from the configurable checkout. * Write CI snapshots off the Docker data disk The builder bind-mounts Docker from /mnt. Reading its image store and writing the compressed archive to that same disk competes for I/O; runner temp keeps the output on the root disk. * Cache CI dependencies separately from workspace source Ordinary source edits invalidate the dependency installation layer because it follows the full checkout copy. Reuse locked third-party dependencies while retaining the complete source installation and its resolution fallback. * Make dependency manifest extraction portable Use portable path-preserving shell commands for the BuildKit manifest stage and cover nested paths containing spaces.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Run dependency manifest extraction with Bash Use the shell required by the manifest extraction stage so the BuildKit RUN command works with Docker's default shell.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden dependency manifest extraction Restore the current UV and prek versions, propagate manifest discovery and copy failures, and cover failure injection in the extraction tests.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address CI snapshot review and separate dependency caching * Name the CI image snapshot artifact v1 There has never been an earlier version of this artifact on main, so the -v2 suffix left over from earlier branch revisions would puzzle the next reader. Snapshot artifacts are scoped to the workflow run that uploads them and kept for two days, so no consumer can see an artifact under the old name and no compatibility shim is needed. A test now checks that the upload and restore steps use the same artifact name, so the two cannot drift apart silently. * Keep CI image snapshots for one day A snapshot is only downloaded by consumer jobs of the run that uploaded it. Once it expires, a re-run restores the image stash instead, which keeps its two-day retention. One day halves the storage the snapshot adds on top of the stash. * Drop snapshot-ci-images assertion per review Co-authored-by: Yossi Eliaz <zozo123@users.noreply.github.com> * Keep CI image snapshots faithful across reruns and check them before download - Archive and extract xattrs in every namespace: GNU tar restores only user.* by default, which dropped overlay2's trusted.overlay.opaque and file capabilities, so a restored image could differ from the built one. - Upload the snapshot with overwrite, plus a small metadata artifact that is uploaded even when the export fails; it reads "unavailable" until the archive is complete, so a rerun never restores an earlier attempt. - Check the metadata against the checkout and daemon before downloading the snapshot, and bound extraction with a timeout that falls back to the stash after cleaning partial state. - Load images for prepare_breeze_and_image and prepare_single_ci_image through one shared load_image action. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Verify restored CI image snapshots and route them through breeze - Record the image ID and a hash of its merged file tree when creating the snapshot and compare both after restoring: a restore that lost an opaque directory marker keeps the image ID but changes the tree, so it now falls back to the stash instead of passing `bash -c true`. - Restore through `breeze ci-image load --snapshot-file`, keeping breeze's environment checks, image check, diagnostics and rebuild marker. - Bound the daemon stop and start as well as the extraction. - Download the snapshot metadata by pattern, which adds no error annotation when the builder made no snapshot. - Limit snapshots to GitHub-hosted runners, where self-hosted runners already restore the stash from their own storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check that an unsupported image store skips pruning and publishes only the unavailable marker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sort the snapshot file-tree listing outside the container Sorting the CI image's 265k-entry listing inside the container spilled to temporary files in /tmp while find was still walking it, so every run listed different files and every consumer rejected the snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Create CI image snapshots through breeze ci-image save The builder now calls `breeze ci-image save --snapshot-file`, mirroring the `breeze ci-image load --snapshot-file` that consumers restore with, so both sides of the snapshot go through breeze and its environment checks. * Retire the snapshot metadata when its archive upload fails and fingerprint ownership and size The export writes metadata for a complete archive before the upload; if the upload then fails, that metadata would point consumers at an earlier attempt's archive. The tree fingerprint now also covers owner, group and size, and the artifact actions use the v7.0.2/v8.0.2 pins used elsewhere in the repository. Generated-by: Claude Opus 5 --------- (cherry picked from commit bf177e2) Co-authored-by: Yossi Eliaz <zozo123@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
1 task done
shahar1
marked this pull request as ready for review
October 31, 2026 21:01
shahar1
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl,
potiuk and
vatsrahul1001
as code owners
October 31, 2026 21:01
This was referenced Nov 1, 2026
vatsrahul1001
pushed a commit
that referenced
this pull request
Nov 4, 2026
…ts (#74173) (#76183) * Restore the CI image from a snapshot of Docker's data directory Every job that prepares the CI image runs `docker image load` on the stash, which unpacks and checksums every layer of an 8 GB image again, about two minutes per job. Extracting a copy of the image store into a stopped daemon gives the same image in about half a minute on the same runners. The image stash stays as it is, for other consumers and as the fallback when a snapshot does not fit the daemon a job runs on. * Recover Docker safely when CI snapshot materialization fails Snapshot failures must not prevent the authoritative stash from loading, and an older branch snapshot must not substitute an image for the current checkout. * Scope CI snapshots to the current workflow run Raw daemon snapshots are job handoffs, so publishing them into branch caches unnecessarily exposes later runs to arbitrary checkout refs. * Publish authoritative caches before optional CI snapshot execution The snapshot script runs checkout code and prunes Docker state. Existing cache publications must finish before that optional execution. * Isolate CI snapshot creation from branch cache publishers The optional snapshot producer executes checkout code without publishing branch caches; its artifacts are available only to consumers in the same workflow run. * Bind optional snapshot jobs to the workflow commit Snapshot producers must not accept arbitrary checkout refs in privileged workflow contexts, and a partially failed Docker stop must still trigger restart cleanup. * Avoid reloading the CI image before snapshot creation Preparing the authoritative image in a second runner adds nearly three minutes before consumers can start. All branch publications must finish before the optional snapshot code prunes Docker state. * Execute the snapshot script from the immutable workflow commit The builder can check out a configurable ref before publishing caches. Snapshot execution must read the workflow commit itself rather than trust files left in that working tree. * Keep optional snapshots in an isolated read-only job Executing snapshot code inside the configurable-ref builder introduces a cache-poisoning finding. A separate job preserves the security boundary; its latency cost must remain explicit when measuring the runner-time benefit. * Preserve Docker state when snapshot inventory checks fail An unavailable daemon must not be mistaken for an empty image store before destructive snapshot materialization. * Retrigger CI after provider compatibility timeout * Create CI snapshots in the builder after publishing caches Reloading the freshly built image in a second job delays every consumer by nearly four minutes. Trusted workflow commands reuse the builder daemon without executing code from the configurable checkout. * Write CI snapshots off the Docker data disk The builder bind-mounts Docker from /mnt. Reading its image store and writing the compressed archive to that same disk competes for I/O; runner temp keeps the output on the root disk. * Cache CI dependencies separately from workspace source Ordinary source edits invalidate the dependency installation layer because it follows the full checkout copy. Reuse locked third-party dependencies while retaining the complete source installation and its resolution fallback. * Make dependency manifest extraction portable Use portable path-preserving shell commands for the BuildKit manifest stage and cover nested paths containing spaces.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Run dependency manifest extraction with Bash Use the shell required by the manifest extraction stage so the BuildKit RUN command works with Docker's default shell.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden dependency manifest extraction Restore the current UV and prek versions, propagate manifest discovery and copy failures, and cover failure injection in the extraction tests.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address CI snapshot review and separate dependency caching * Name the CI image snapshot artifact v1 There has never been an earlier version of this artifact on main, so the -v2 suffix left over from earlier branch revisions would puzzle the next reader. Snapshot artifacts are scoped to the workflow run that uploads them and kept for two days, so no consumer can see an artifact under the old name and no compatibility shim is needed. A test now checks that the upload and restore steps use the same artifact name, so the two cannot drift apart silently. * Keep CI image snapshots for one day A snapshot is only downloaded by consumer jobs of the run that uploaded it. Once it expires, a re-run restores the image stash instead, which keeps its two-day retention. One day halves the storage the snapshot adds on top of the stash. * Drop snapshot-ci-images assertion per review * Keep CI image snapshots faithful across reruns and check them before download - Archive and extract xattrs in every namespace: GNU tar restores only user.* by default, which dropped overlay2's trusted.overlay.opaque and file capabilities, so a restored image could differ from the built one. - Upload the snapshot with overwrite, plus a small metadata artifact that is uploaded even when the export fails; it reads "unavailable" until the archive is complete, so a rerun never restores an earlier attempt. - Check the metadata against the checkout and daemon before downloading the snapshot, and bound extraction with a timeout that falls back to the stash after cleaning partial state. - Load images for prepare_breeze_and_image and prepare_single_ci_image through one shared load_image action. * Verify restored CI image snapshots and route them through breeze - Record the image ID and a hash of its merged file tree when creating the snapshot and compare both after restoring: a restore that lost an opaque directory marker keeps the image ID but changes the tree, so it now falls back to the stash instead of passing `bash -c true`. - Restore through `breeze ci-image load --snapshot-file`, keeping breeze's environment checks, image check, diagnostics and rebuild marker. - Bound the daemon stop and start as well as the extraction. - Download the snapshot metadata by pattern, which adds no error annotation when the builder made no snapshot. - Limit snapshots to GitHub-hosted runners, where self-hosted runners already restore the stash from their own storage. * Check that an unsupported image store skips pruning and publishes only the unavailable marker * Sort the snapshot file-tree listing outside the container Sorting the CI image's 265k-entry listing inside the container spilled to temporary files in /tmp while find was still walking it, so every run listed different files and every consumer rejected the snapshot. * Create CI image snapshots through breeze ci-image save The builder now calls `breeze ci-image save --snapshot-file`, mirroring the `breeze ci-image load --snapshot-file` that consumers restore with, so both sides of the snapshot go through breeze and its environment checks. * Retire the snapshot metadata when its archive upload fails and fingerprint ownership and size The export writes metadata for a complete archive before the upload; if the upload then fails, that metadata would point consumers at an earlier attempt's archive. The tree fingerprint now also covers owner, group and size, and the artifact actions use the v7.0.2/v8.0.2 pins used elsewhere in the repository. Generated-by: Claude Opus 5 --------- (cherry picked from commit bf177e2) Co-authored-by: Yossi Eliaz <zozo123@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every job that prepares the CI image runs
docker image loadon the stash,which unpacks and checksums every layer of an 8 GB image again, about two
minutes per job. Extracting a copy of the image store into a stopped daemon
gives the same image in about half a minute on the same runners. The image
stash stays as it is, for other consumers and as the fallback when a
snapshot does not fit the daemon a job runs on.
Snapshot failures must not prevent the authoritative stash from loading, and an older branch snapshot must not substitute an image for the current checkout.
Raw daemon snapshots are job handoffs, so publishing them into branch caches unnecessarily exposes later runs to arbitrary checkout refs.
The snapshot script runs checkout code and prunes Docker state. Existing cache publications must finish before that optional execution.
The optional snapshot producer executes checkout code without publishing branch caches; its artifacts are available only to consumers in the same workflow run.
Snapshot producers must not accept arbitrary checkout refs in privileged workflow contexts, and a partially failed Docker stop must still trigger restart cleanup.
Preparing the authoritative image in a second runner adds nearly three minutes before consumers can start. All branch publications must finish before the optional snapshot code prunes Docker state.
The builder can check out a configurable ref before publishing caches. Snapshot execution must read the workflow commit itself rather than trust files left in that working tree.
Executing snapshot code inside the configurable-ref builder introduces a cache-poisoning finding. A separate job preserves the security boundary; its latency cost must remain explicit when measuring the runner-time benefit.
An unavailable daemon must not be mistaken for an empty image store before destructive snapshot materialization.
Retrigger CI after provider compatibility timeout
Create CI snapshots in the builder after publishing caches
Reloading the freshly built image in a second job delays every consumer by nearly four minutes. Trusted workflow commands reuse the builder daemon without executing code from the configurable checkout.
The builder bind-mounts Docker from /mnt. Reading its image store and writing the compressed archive to that same disk competes for I/O; runner temp keeps the output on the root disk.
Ordinary source edits invalidate the dependency installation layer because it follows the full checkout copy. Reuse locked third-party dependencies while retaining the complete source installation and its resolution fallback.
Use portable path-preserving shell commands for the BuildKit manifest stage and cover nested paths containing spaces.\n\nCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Use the shell required by the manifest extraction stage so the BuildKit RUN command works with Docker's default shell.\n\nCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Restore the current UV and prek versions, propagate manifest discovery and copy failures, and cover failure injection in the extraction tests.\n\nCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Address CI snapshot review and separate dependency caching
Name the CI image snapshot artifact v1
There has never been an earlier version of this artifact on main, so
the -v2 suffix left over from earlier branch revisions would puzzle the
next reader. Snapshot artifacts are scoped to the workflow run that
uploads them and kept for two days, so no consumer can see an artifact
under the old name and no compatibility shim is needed.
A test now checks that the upload and restore steps use the same
artifact name, so the two cannot drift apart silently.
A snapshot is only downloaded by consumer jobs of the run that uploaded
it. Once it expires, a re-run restores the image stash instead, which
keeps its two-day retention. One day halves the storage the snapshot
adds on top of the stash.
Co-authored-by: Yossi Eliaz zozo123@users.noreply.github.com
user.* by default, which dropped overlay2's trusted.overlay.opaque and
file capabilities, so a restored image could differ from the built one.
is uploaded even when the export fails; it reads "unavailable" until the
archive is complete, so a rerun never restores an earlier attempt.
the snapshot, and bound extraction with a timeout that falls back to
the stash after cleaning partial state.
through one shared load_image action.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
snapshot and compare both after restoring: a restore that lost an opaque
directory marker keeps the image ID but changes the tree, so it now falls
back to the stash instead of passing
bash -c true.breeze ci-image load --snapshot-file, keeping breeze'senvironment checks, image check, diagnostics and rebuild marker.
when the builder made no snapshot.
already restore the stash from their own storage.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
Sorting the CI image's 265k-entry listing inside the container spilled
to temporary files in /tmp while find was still walking it, so every run
listed different files and every consumer rejected the snapshot.
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com
The builder now calls
breeze ci-image save --snapshot-file, mirroring thebreeze ci-image load --snapshot-filethat consumers restore with, so bothsides of the snapshot go through breeze and its environment checks.
The export writes metadata for a complete archive before the upload; if the upload then fails, that metadata would point consumers at an earlier attempt's archive. The tree fingerprint now also covers owner, group and size, and the artifact actions use the v7.0.2/v8.0.2 pins used elsewhere in the repository.
Generated-by: Claude Opus 5
(cherry picked from commit bf177e2)
Co-authored-by: Yossi Eliaz zozo123@users.noreply.github.com
Co-authored-by: Cursor Agent cursoragent@cursor.com
Co-authored-by: Claude Opus 5.5 noreply@anthropic.com
Co-authored-by: Shahar Epstein 60007259+shahar1@users.noreply.github.com
Co-authored-by: Jarek Potiuk jarek@potiuk.com