Repository navigation
[v3-4-test] Pin the hardened base images to digests the CI has passed on (#76023) - #76038
Merged
Merged
Conversation
* Refresh the hardened base image mirror every six hours Docker rebuilds the hardened images in place and moves the dhi.io apt repository on at the same time. With a weekly mirror, the base images Airflow builds on lag the repository by up to a week, and any apt install whose dependencies were rebuilt in between fails: build-essential currently cannot be installed because the mirrored base has an older libbz2 than the bzip2 the repository offers (docker-hardened-images/catalog#689). Mirroring copies manifests between registries without pulling layers, so running it several times a day is cheap. Generated-by: Claude Opus 5 * Pin the hardened base images to digests the CI has passed on Building on the mirror's moving tags made CI non-reproducible: any refresh of the mirror changed the image every build started from, without a change in the repository and without the full CI having run on it. A CI run could pass or fail depending only on when it ran, and rebuilding the same commit later gave a different image. The images now build on a digest, and only the upgrade check moves it, in a PR that must pass the full CI before the new base is used. The mirror refreshes every six hours and triggers that upgrade check when it moves a tag past its pin, so the pins follow Docker's rebuilds - and the dhi.io apt repository moving with them - within hours rather than drift behind it. Every pinned and mirrored digest keeps a tag of its own, so a pinned image never becomes untagged. The hardened images keep stock Debian's apt sources beside dhi.io, and apt took whichever offered the higher version, mixing Debian builds into an image whose dependencies were built by Docker. dhi.io is now preferred whenever it publishes the package. Generated-by: Claude Opus 5 * Let dhi.io's package versions win over the image's own Docker renumbered its builds ("1.0.8-6+dhi2" became "1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved to the new numbers before its images did. Every package depending on a renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is then uninstallable, because apt will not replace the image's higher-numbered copy with the repository's lower-numbered one, and the CI image build fails (docker-hardened-images/catalog#689). Every package in the image comes from dhi.io, so letting its repository decide their versions is safe, and it keeps the build working through such a renumbering. Generated-by: Claude Opus 5 * Move the base images to Docker's current Python patch releases Docker stops rebuilding a Python patch release once it publishes the next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old package numbering that the dhi.io apt repository has moved past, and installing packages on them fails (docker-hardened-images/catalog#689). Docker fixed it in the images of the current patch releases. Generated with the Python part of the upgrade check, which also pins the new images to the digests the mirror serves for them. Generated-by: Claude Opus 5 * Compare the registry host exactly in the mirror test Generated-by: Claude Opus 5 * Lower the dhi.io apt pin below 1000 so dependencies can resolve At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on. * Rank the PostgreSQL apt repository above the dhi.io preference install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client. * Use US spelling favors to pass the docs spellcheck The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments. --------- Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> (cherry picked from commit 4843535)
boring-cyborg
Bot
added
area:dev-tools
area:docker-tests
area:production-image
vatsrahul1001
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jedcunningham,
jscheffl,
shahar1 and
vincbeck
as code owners
October 30, 2026 14:17
vatsrahul1001
requested review from
choo121600,
ephraimbuddy and
jason810496
as code owners
October 30, 2026 14:17
shahar1
approved these changes
Oct 30, 2026
Contributor
RunsOn re-runs them automatically when the run finishes; no action is needed. |
vatsrahul1001
enabled auto-merge (squash)
October 30, 2026 14:45
potiuk
approved these changes
Oct 30, 2026
vatsrahul1001
disabled auto-merge
October 30, 2026 15:53
vatsrahul1001
added a commit
that referenced
this pull request
Oct 30, 2026
…#76038) * Refresh the hardened base image mirror every six hours Docker rebuilds the hardened images in place and moves the dhi.io apt repository on at the same time. With a weekly mirror, the base images Airflow builds on lag the repository by up to a week, and any apt install whose dependencies were rebuilt in between fails: build-essential currently cannot be installed because the mirrored base has an older libbz2 than the bzip2 the repository offers (docker-hardened-images/catalog#689). Mirroring copies manifests between registries without pulling layers, so running it several times a day is cheap. Generated-by: Claude Opus 5 * Pin the hardened base images to digests the CI has passed on Building on the mirror's moving tags made CI non-reproducible: any refresh of the mirror changed the image every build started from, without a change in the repository and without the full CI having run on it. A CI run could pass or fail depending only on when it ran, and rebuilding the same commit later gave a different image. The images now build on a digest, and only the upgrade check moves it, in a PR that must pass the full CI before the new base is used. The mirror refreshes every six hours and triggers that upgrade check when it moves a tag past its pin, so the pins follow Docker's rebuilds - and the dhi.io apt repository moving with them - within hours rather than drift behind it. Every pinned and mirrored digest keeps a tag of its own, so a pinned image never becomes untagged. The hardened images keep stock Debian's apt sources beside dhi.io, and apt took whichever offered the higher version, mixing Debian builds into an image whose dependencies were built by Docker. dhi.io is now preferred whenever it publishes the package. Generated-by: Claude Opus 5 * Let dhi.io's package versions win over the image's own Docker renumbered its builds ("1.0.8-6+dhi2" became "1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved to the new numbers before its images did. Every package depending on a renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is then uninstallable, because apt will not replace the image's higher-numbered copy with the repository's lower-numbered one, and the CI image build fails (docker-hardened-images/catalog#689). Every package in the image comes from dhi.io, so letting its repository decide their versions is safe, and it keeps the build working through such a renumbering. Generated-by: Claude Opus 5 * Move the base images to Docker's current Python patch releases Docker stops rebuilding a Python patch release once it publishes the next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old package numbering that the dhi.io apt repository has moved past, and installing packages on them fails (docker-hardened-images/catalog#689). Docker fixed it in the images of the current patch releases. Generated with the Python part of the upgrade check, which also pins the new images to the digests the mirror serves for them. Generated-by: Claude Opus 5 * Compare the registry host exactly in the mirror test Generated-by: Claude Opus 5 * Lower the dhi.io apt pin below 1000 so dependencies can resolve At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on. * Rank the PostgreSQL apt repository above the dhi.io preference install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client. * Use US spelling favors to pass the docs spellcheck The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments. --------- Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> (cherry picked from commit 4843535) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #76023 to
v3-4-test. The auto-backport failed to apply cleanly, so this is a manual backport.Pins the hardened base images to digests CI has passed on (ending the mirror-tag drift that was reddening
v3-4-testCI) and adds thebreeze workflow-run upgrade-check/mirror-base-imagesrelease-process step.Conflict resolution: the only cherry-pick conflict was in the generated
output_ci_upgrade.txt; the four generated breeze help-images (output_ci_upgrade.*,output_workflow-run_upgrade-check.*) were regenerated onv3-4-testso their hashes match this branch's command definitions. Full local prek suite passes.(cherry picked from commit 4843535)
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines