[youtalk]

• fix(ansible): make locked mode deliver a fully frozen closure #7237

Description

Follows #7237, which fixed locked mode's dependency-closure logic; this PR adds the CI lanes that actually exercise the locked install path. No workflow had ever run with use_locked_versions=true / USE_LOCKFILE=true, so a locked-mode regression (or an unresolvable package like ros-build-essential) was invisible to CI and every fix in #7237 rested on a local manual harness.

The lanes worked immediately, and the first thing they caught was not a contributor's regression but the archive's own decay — see "The blocker the lane found" below, which is why this PR now also carries an ansible/ fix.

What changed

• health-check-ansible.yaml: a locked: [false, true] matrix axis on install-dev-env, so the job runs 4 cells (ubuntu:22.04 and ubuntu:24.04 × unlocked/locked), passed to the playbook as -e use_locked_versions=${{ matrix.locked }} and reflected in the disk-usage step-summary heading.
• validate-lockfiles.yaml: a build-locked-base job that bakes the locked base image for humble and jazzy with USE_LOCKFILE=true, behind the same run:health-check label the repo already uses for its other expensive lanes, plus a build-locked-base-required aggregator so the lane has one stable name a branch ruleset could require. The bake gates on its own changed-files job wired through run-condition, so the workflow trigger no longer has to serve both an 8-second lint and two cold container builds with one filter.
• ansible/roles/version_lock/: a dated snapshot.ubuntu.com apt source in locked mode, the Ubuntu-archive counterpart of the dated snapshots.ros.org source the ros2 role already configures. Lockfiles gain ubuntu_snapshot_date; the generator requires and validates it.

The blocker the lane found

An exact-version APT pin only binds a version apt can still reach. Ubuntu drops a superseded build from <suite>-updates and <suite>-security as soon as it publishes the replacement, so a pin taken at lockfile-generation time stops matching anything the moment a security update lands — apt installs the current candidate and version_lock's drift assert fails the play.

That happened on its own, with no change to this repository: 17 days after the lockfiles were last regenerated, wget had moved on both distros (1.21.2-2ubuntu1.4 → .5 on jammy, 1.21.4-1ubuntu4.4 → .5 on noble) and took both locked cells with it. Since install-dev-env-required is a required check, leaving this unfixed would have made a required check that breaks on unrelated work — exactly the kind people learn to ignore.

snapshot.ubuntu.com republishes the pockets as they stood at a given timestamp, so pointing at the lockfile's own date makes every pinned build downloadable again. At 20260805T000000Z — the date the current lockfiles were taken — it serves 25/25 apt_pins for jammy and 25/25 for noble, on both amd64 and arm64, from one tree, signed by the stock ubuntu-archive-keyring.

The source is added alongside the distribution's own sources, not in place of them. The pins remain the contract: at Pin-Priority: 1001 they select the locked build wherever it is served, so this source only has to keep that build reachable. install_dev_env also runs on contributors' own machines, where rewriting /etc/apt/sources.list is not this role's to do, and packages outside apt_pins keep resolving exactly as they do today. A newly introduced Ubuntu-archive package that nobody pinned is already a hard failure via check_pin_completeness.py.

Notes for reviewers

• The new axis renames the matrix checks: install-dev-env (ubuntu:22.04) becomes (ubuntu:22.04, false) and (ubuntu:22.04, true). install-dev-env-required is unchanged and remains the branch-protection anchor; the branch ruleset on main requires that name and no raw matrix cell name, so the rename breaks no gate.
• build-locked-base now has an aggregator, but promoting it to a required check is a ruleset change and not part of this PR. Until someone makes it, the lane that works today is advisory while the required lane (install-dev-env-required) is the one archive decay breaks.
• CI cost: the locked cells took 18m43s and 16m57s against 18m17s and 11m39s unlocked. Most of the locked delta is the first apt-get update also fetching the snapshot indices, about 48 MB on jammy and 32 MB on noble; a dated snapshot never changes, so later updates are cheap. Each build-locked-base cell moved from 1m41s/2m8s to 1m55s/2m26s.

Known gaps, all out of scope here

• arm64 is uncovered. Two of the four lockfiles carry real pins and ubuntu-24.04-arm runners are already in use in health-check.yaml, so this is a gap of coverage rather than of capability.
• The locked core, universe and base-cuda stages are uncovered. Only base is baked.
• No scheduled lane. This PR catches a regression a contributor writes; it cannot catch the one that arrives on its own, and the wget failure above was the second kind. A weekly read-only cron over the two bakes would cover most of that decay cheaply. The comment in regenerate-lockfiles.yaml records a prior rejection of calendar cron, but that was about auto-changing lockfiles rather than a read-only verify. Proposed as a follow-up.

Related: #6862

How was this PR tested?

Staged on youtalk#232, whose tree over every file this PR touches is byte-identical to the head here (cb36c14). Since the point of this PR is the lanes themselves, the evidence is that they really run rather than silently no-op, and that they are green on a tree where the blocker above is fixed.

install-dev-env (ubuntu:22.04, true)   pass  18m43s
install-dev-env (ubuntu:24.04, true)   pass  16m57s
install-dev-env (ubuntu:22.04, false)  pass  18m17s
install-dev-env (ubuntu:24.04, false)  pass  11m39s
install-dev-env-required               pass      3s
build-locked-base (humble)             pass  1m55s
build-locked-base (jazzy)              pass  2m26s
build-locked-base-required             pass      6s
validate-lockfiles                     pass      8s

• health-check-ansible — all four cells green plus install-dev-env-required, and the locked ones genuinely execute the version_lock asserts (All 71 pinned packages match the lockfile., ROS closure matches the 2026-05-14/2026-04-13 snapshot., All newly-installed Ubuntu packages are covered by apt_pins., failed=0).
• validate-lockfiles — build-locked-base (humble) and (jazzy) both green, and the new changed-files → require-label → build-locked-base-required chain reports as intended. Each bake is genuinely cold (no cache-from, fresh builder) and its ansible layer shows the reconcile reporting changed and all asserts ok rather than skipping.
• health-check-pr — the docker-build variants are unaffected by the trigger changes.

The ansible/ fix was additionally reproduced outside CI, running the cell's own command (install_dev_env --skip-tags artifacts -e use_locked_versions=true) in a bare ubuntu:22.04 and ubuntu:24.04 container: ok=146 changed=66 failed=0 and ok=138 changed=62 failed=0 respectively. apt-cache policy wget in the finished jammy container shows the mechanism directly — the live archive offers only 1.21.2-2ubuntu1.5 at priority 500, while the pinned .4 comes from the snapshot at priority 1001.

This is the first time any workflow in this repository has run with use_locked_versions / USE_LOCKFILE set.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[youtalk]

@xmfcx I've rebased the branch. It's ready to review.

[xmfcx]

Thank you for this. Locked mode had no CI at all before this PR, and that gap is real: nothing on main sets use_locked_versions or USE_LOCKFILE in any workflow. Both lanes do what the description says, and I confirmed that they really exercise the locked path rather than no-op. The two-lane split is also worth keeping, because each lane covers something the other cannot.

I re-ran the checks against the current head today. There is one blocker. Everything else is a suggestion or a nit.

First, your open question

The matrix rename is safe. The branch ruleset on main requires install-dev-env-required, and no raw matrix cell name is a required context. A repo-wide grep for install-dev-env (ubuntu: finds nothing. So the rename from install-dev-env (ubuntu:22.04) to (ubuntu:22.04, false) and (ubuntu:22.04, true) breaks no gate.

The blocker: both locked cells are red on this PR's own run

install-dev-env (ubuntu:22.04, true)   fail  17m3s   wget: locked 1.21.2-2ubuntu1.4, installed 1.21.2-2ubuntu1.5
install-dev-env (ubuntu:24.04, true)   fail  10m39s  wget: locked 1.21.4-1ubuntu4.4, installed 1.21.4-1ubuntu4.5
install-dev-env (ubuntu:22.04, false)  pass  17m45s
install-dev-env (ubuntu:24.04, false)  pass  12m45s
install-dev-env-required               fail  3s      install-dev-env: failure

wget is the only entry in version_lock_drift on either cell. Both cells report failed=1. That is a clean 2x2 result, and it is exactly the kind of regression the PR set out to catch. The lane works.

The cause is not the PR. It is the calendar. ros2_dev_tools installs wget with state: latest, and --skip-tags artifacts does not skip that role. Ubuntu then superseded both pinned builds, and it drops superseded versions from the pockets. An APT preference that names a version the archive no longer serves matches nothing, so apt installs the current candidate and the drift assert fires.

I reproduced it outside CI:

$ docker run --rm ubuntu:24.04 apt-cache madison wget
      wget | 1.21.4-1ubuntu4.5 | noble-updates/main amd64 Packages
      wget | 1.21.4-1ubuntu4.5 | noble-security/main amd64 Packages
      wget | 1.21.4-1ubuntu4   | noble/main amd64 Packages

Today this is one stale pin, not a pile of them. I checked all 25 apt_pins of each distro against the live archive, and wget is the only unreachable entry on either. The concern is the mechanism, not the count. The lockfiles were last regenerated on 2026-08-05, and the lane broke 17 days later.

Why the bake lane stays green, and why that is not a counter-example

install_rmw runs the rmw_implementation role only, so the bake never installs wget. Of the 25 apt_pins, the pinned ros:jazzy-ros-base image ships nine (build-essential, cmake, git, python3-setuptools and five python3-flake8-* packages). The build adds pipx and python3-apt from the live archive. All eleven match the lockfile today, and the drift check skips the other 14 pins and every nvidia_pins entry, because the bake installs none of them.

I built the locked jazzy base locally on 2026-08-22 to confirm this. It passed with ok=25 changed=8 failed=0 in 151s, on the same commit whose ansible cells are red.

So the bake passes because its package footprint is small, not because the base-image digest froze wget.

Why an image digest does not fix this

Pinning the health-check container by digest looks like the natural fix, and it does not work. wget is absent from the ubuntu:24.04 image, so the role downloads it from the live archive at install time. In fact ubuntu:24.04 ships none of the 25 apt_pins. An image digest freezes the preinstalled package set, not what apt can fetch later.

What does fix it

Point apt at a dated Ubuntu archive, the same way the role already points ROS at a dated snapshot in snapshot_source.yaml. snapshot.ubuntu.com serves the exact pinned build at the lockfile's own date:

$ curl -s https://snapshot.ubuntu.com/ubuntu/20260805T000000Z/dists/noble-updates/main/binary-amd64/Packages.gz \
    | gunzip | awk '/^Package: wget$/{f=1} f&&/^Version:/{print $2; exit}'
1.21.4-1ubuntu4.4

noble-security serves the same version, and jammy-updates answers at the same timestamp for humble. This is the only option I found that freezes what apt can download. Nothing in the repo uses it yet, so it is real work, and I have not wired it into the role and run it. Two weaker alternatives, for completeness:

• Regenerate the lockfiles in this PR, and state in the body that the lane needs a refresh on every Ubuntu security update to a pinned package.
• Exempt packages whose Ubuntu build floats, in the style of the KERNEL_COUPLED and OUT_OF_SCOPE lists in check_pin_completeness.py. This weakens the guarantee the role exists to give, so I put it last.

This matters because install-dev-env-required is a required check. Without the run:health-check label, install-dev-env is skipped and the aggregator exits 0. But an ansible PR without the label already goes red on require-label, so a maintainer adds the label and then meets the stale pin. A required check that breaks on work unrelated to the pin is a check people learn to ignore, and that would undo what this PR is for.

Suggestions

Consider moving build-locked-base into health-check-ansible.yaml. That workflow already owns changed-files, a require-label job wired to run-condition, the same label, and a stable aggregator. A bake of the locked base is a validation of the lockfile, so validate-lockfiles is a defensible home by name. The argument for moving is not the name, it is the shared paths: gate: one filter now serves both an 8-second lint and two cold container builds, and those two want opposite filters. Over the last 120 days the old filter matched 6 commits on main and the new one matches 32, but only 6 of those 32 touch a file the base target consumes.

Consider a scheduled lane, here or as a follow-up. This PR catches a regression that a contributor writes. It cannot catch the more common one that arrives on its own, and the wget failure is that second kind. A weekly cron over the two bakes, at about two minutes each, would cover most of the decay for very little. The comment in regenerate-lockfiles.yaml records a prior rejection of calendar cron, but that was about auto-changing lockfiles, not about a read-only verify.

Please refresh the evidence in the body. The linked runs are from 2026-07-28 at head 4bf96b588, and the head is now 8f60853c7. So the "same head commit as this PR" line and "all 23 checks green" no longer hold.

Two gaps worth naming in the body, both out of scope here. arm64 is uncovered, although two of the four lockfiles carry real pins and ubuntu-24.04-arm runners are already in use in health-check.yaml. The locked core, universe and base-cuda stages are uncovered too. I also noticed build-locked-base emits two cell names with no aggregator, so it cannot become a required check as it stands. You already flag the missing aggregator, and it is worth saying that the lane which works today is advisory while the required lane is the one archive decay breaks.

Left the rest inline. Thanks again for picking this up, the premise is right and the two-minute bake is a good deal for coverage nothing else provides.

---

Written with Claude Code, from the diff and the live CI runs. Every claim was checked against the PR, the job logs, the branch ruleset and the Ubuntu archives, and I built the locked jazzy base locally to confirm the bake result. I have not run the snapshot.ubuntu.com option end to end.

[xmfcx]

This call site has no run-condition, and that makes it behave differently from the three that already exist.

The reusable workflow guards its inner job with if: ${{ inputs.run-condition }}, which defaults to true, and runs exit 1 when the label is absent. The other three call sites wire that input to a changed-files job, so they report skipped on an unrelated PR. For example, health-check-ansible.yaml. The same pattern is in health-check-ansible-artifacts.yaml and health-check-pr.yaml.

All four emit the same context, require-label / require-label, and that context is required on main. Most of the pressure is pre-existing, because health-check-pr.yaml already watches ansible/** and docker/**. The genuinely new case is narrow: a PR that touches only .github/workflows/validate-lockfiles.yaml matches no other workflow's changed-files, so the three siblings report skipped and this one reports red.

I am not certain that red blocks the merge, because this PR carries four check runs under that one name and GitHub resolves the context from the most recent of them. Either way, a red check on a workflow-only PR reads as a regression. If the job moves into health-check-ansible.yaml, this disappears for free.

[youtalk]

Fixed. changed-files job added and wired through run-condition, so this call site now behaves like the three in health-check-ansible.yaml, health-check-ansible-artifacts.yaml and health-check-pr.yaml: skipped on a PR that reaches the workflow without touching a bake input, rather than red. On the verification branch it reports pass in 3s alongside its siblings.

On resolving it by moving the job into health-check-ansible.yaml: I looked at that first and it does not work as a straight move. That workflow's changed-files watches ansible/**, ansible-galaxy-requirements.yaml and its own file — not docker/*.Dockerfile and not docker/docker-bake.hcl, both of which the base target consumes. The bake would stop running on a Dockerfile change, and widening the filter to fix that would start install-dev-env on Docker-only PRs, which touch no Ansible. Giving the bake its own changed-files here gets the same decoupling without trading one wrong trigger for another.

[youtalk]

Thank you — the diagnosis is exactly right, and I reproduced it before changing anything. The live archive serves only 1.21.2-2ubuntu1.5 / 1.21.4-1ubuntu4.5; neither pinned .4 build exists any more, so the pin matches nothing.

The fix

snapshot.ubuntu.com, as you proposed, wired into version_lock as tasks/ubuntu_snapshot_source.yaml. The lockfiles gain ubuntu_snapshot_date; the generator requires it and the validator rejects a lockfile without it. At 20260805T000000Z — the date the current lockfiles were taken — the snapshot serves 25/25 apt_pins for jammy and 25/25 for noble, on both architectures from one tree, verified against the live service.

One deviation: I added the source alongside the distribution's own sources rather than replacing them. The pins stay the contract — at priority 1001 they select the locked build wherever it is served, so the snapshot only has to keep it reachable. install_dev_env also runs on contributors' machines, where rewriting /etc/apt/sources.list is not this role's to do, and an unpinned package is already a hard failure via check_pin_completeness.py. It is a weaker freeze than you described; say the word and I will switch it to replacement.

Evidence

install-dev-env (ubuntu:22.04, true)   pass  18m43s
install-dev-env (ubuntu:24.04, true)   pass  16m57s
install-dev-env-required               pass      3s
build-locked-base (humble)/(jazzy)     pass  1m55s / 2m26s

Staged on youtalk#232, plus a local run of the cell's own command in both bare containers (failed=0). Details in the PR body, which is also refreshed against the current head and now names the arm64 and core/universe/base-cuda gaps.

Your suggestions

Moving the bake into health-check-ansible.yaml — not taken. Its changed-files does not watch docker/*.Dockerfile or docker/docker-bake.hcl, which the base target consumes, and widening it would start install-dev-env on Docker-only PRs. The bake now has its own changed-files wired through run-condition, which dissolves the shared-filter problem without the coupling. Aggregator and v4 are in.

A scheduled lane — agreed, including that the earlier cron rejection was about auto-changing lockfiles rather than a read-only verify. I would rather land it as a follow-up, but happy to add it here if you prefer.

[xmfcx]

The new required timestamp breaks the lockfile regeneration workflow.

I reproduced the missing-input failure in Ubuntu 22.04 and 24.04 containers on amd64. A disposable package fixture passes with the base generator and fails with this PR. Supplying UBUNTU_SNAPSHOT_DATE lets the PR generator and the copied workflow shell step finish on that fixture.

Earlier workflow lint, lockfile validation, and snapshot installation tests passed on this same commit. The snapshot tests covered package download, repeated installation, and cleanup on both Ubuntu versions.

The full regeneration workflow and arm64 runtime tests did not run. The fixture does not prove package availability at the supplied timestamp.

AI usage: Codex reviewed the diff, callers, discussion, and CI evidence, ran focused local tests, and challenged the finding in two further review passes.

[xmfcx]

Supply the new required timestamp in the regeneration workflow

This guard makes UBUNTU_SNAPSHOT_DATE mandatory, but .github/workflows/regenerate-lockfiles.yaml:88-94 still supplies only ROS_DISTRO and ROS_SNAPSHOT_DATE. Both dispatch modes use that step, so each matrix cell exits here without generating a lockfile.

Please supply UBUNTU_SNAPSHOT_DATE for both workflow_dispatch and repository_dispatch, using a timestamp that matches the recorded package set.

I reproduced exit status 1 in both Ubuntu CI images on amd64 with only the two ROS variables that the step supplies:

Error: UBUNTU_SNAPSHOT_DATE is not set (snapshot.ubuntu.com timestamp, e.g. 20260805T000000Z).

The failure occurs before architecture detection. In a disposable package fixture, supplying the variable lets the copied workflow shell step finish. The full regeneration workflow did not run.