Skip to content

[v3-4-test] Run dependency explanations separately from main AMD canaries (#74311) #76182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
potiuk merged 1 commit into from
Oct 31, 2026
Merged

[v3-4-test] Run dependency explanations separately from main AMD canaries (#74311) #76182

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
254 changes: 254 additions & 0 deletions .github/workflows/dependency-upgrade-report.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
---
name: Dependency upgrade report
# Both the job gate and the API validation reject PRs and foreign repositories. Only
# completed main schedule/dispatch runs are executed, with a read-only token and no secrets.
"on": # zizmor: ignore[dangerous-triggers]
workflow_run:
workflows: ["Tests (AMD)"]
types: [completed]
branches: [main]
permissions:
contents: read
actions: read
# Every canary attempt fires this workflow and gets its own report of the inputs it saved.
concurrency:
group: dependency-report-${{ github.event.workflow_run.id }}-${{ github.event.workflow_run.run_attempt }}
cancel-in-progress: false
jobs:
plan:
runs-on: ubuntu-slim
timeout-minutes: 5
if: >-
github.event.workflow_run.head_repository.full_name == github.repository &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_dispatch')
outputs:
matrix: ${{ steps.plan.outputs.matrix }}
has-reports: ${{ steps.plan.outputs.has-reports }}
source-sha: ${{ steps.plan.outputs.source-sha }}
source-run-id: ${{ steps.plan.outputs.source-run-id }}
steps:
- name: "Select inputs from the original canary run"
id: plan
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
SOURCE_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
with:
script: |
const rawId = process.env.SOURCE_RUN_ID;
if (!/^[1-9][0-9]*$/.test(rawId)) throw new Error('Invalid source run ID');
const run_id = Number(rawId);
if (!Number.isSafeInteger(run_id)) throw new Error('Invalid source run ID');
const rawAttempt = process.env.SOURCE_RUN_ATTEMPT;
if (!/^[1-9][0-9]{0,3}$/.test(rawAttempt)) throw new Error('Invalid source run attempt');
const attempt_number = Number(rawAttempt);
const repo = context.repo;
const { data: run } = await github.rest.actions.getWorkflowRunAttempt(
{ ...repo, run_id, attempt_number });
if (run.head_repository.full_name !== `${repo.owner}/${repo.repo}` ||
run.head_branch !== 'main' || run.path !== '.github/workflows/ci-amd.yml' ||
!['schedule', 'workflow_dispatch'].includes(run.event) || run.status !== 'completed') {
throw new Error('Only completed main AMD canary runs are supported');
}
const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts,
{ ...repo, run_id, per_page: 100 });
const include = [];
const skipped = [];
const explained = [];
const inputName = new RegExp('^dependency-report-inputs-(3\\.[0-9]+)-' +
'(constraints(?:-source-providers|-no-providers)?)-(true|false)-([1-9][0-9]*)$');
const saved = artifacts.map(a => [a, inputName.exec(a.name)]).filter(([, match]) => match);
// Only RunsOn runners can read RunsOn's S3 cache, where a canary on RunsOn keeps its CI image.
const runsOnRunner = `runs-on=${process.env.GITHUB_RUN_ID}/family=m8a+m8i+m7a+m7i+m6i` +
'/image=ubuntu22-full-x64/volume=150gb/spot=co/cpu=8/ram=32';
for (const [input, [, python, mode, useUv, attempt]] of saved) {
if (attempt !== rawAttempt) continue;
const entry = `${python}:${mode}`;
// A re-run canary saves its inputs again; they are explained only for the first attempt.
if (saved.some(([, m]) => m[1] === python && m[2] === mode && m[3] === useUv &&
Number(m[4]) < attempt_number)) {
explained.push(entry);
continue;
}
// The stash action names the artifact after the stash key, with '/' as '_', and the branch.
// On RunsOn runners the image itself goes to RunsOn's S3 cache and the artifact is only
// its digest, stashed under the key plus '-s3' (see .github/actions/stash-save).
const imageName = `ci-image-save-v3-linux_amd64-${python}-main`;
const image = artifacts.find(a => a.name === imageName) ||
artifacts.find(a => a.name === `ci-image-save-v3-linux_amd64-${python}-s3-main`);
const missing = input.expired ? 'inputs expired' :
!image ? 'CI image missing' : image.expired ? 'CI image expired' : '';
if (missing) {
core.warning(`Skipping ${entry}: original inputs or CI image unavailable (${missing})`);
skipped.push(`${entry} (${missing})`);
continue;
}
const fromS3 = image.name !== imageName;
include.push({ python, mode, 'use-uv': useUv, 'inputs-id': input.id, 'image-id': image.id,
'image-source': fromS3 ? 's3' : 'artifact', runner: fromS3 ? runsOnRunner : 'ubuntu-22.04' });
}
core.setOutput('matrix', JSON.stringify({ include }));
core.setOutput('has-reports', include.length > 0 ? 'true' : 'false');
core.setOutput('source-sha', run.head_sha);
core.setOutput('source-run-id', rawId);
if (!include.length && skipped.length && Number(process.env.GITHUB_RUN_ATTEMPT) > 1) {
throw new Error('Saved report inputs are missing or expired');
}
if (!include.length && !skipped.length) {
core.notice(explained.length ?
'The saved report inputs were already saved by an earlier canary attempt' :
'No saved report inputs: this run did not request independent diagnostics');
}
await core.summary.addHeading('Dependency upgrade diagnostics')
.addLink(`Source canary run (attempt ${rawAttempt})`, run.html_url)
.addRaw(`\n\nSource revision: ${run.head_sha}\n\nReports: ${include.length}\n`)
.addRaw(skipped.length ? `\nSkipped: ${skipped.join(', ')}\n` : '')
.addRaw(explained.length ? `\nSaved by an earlier attempt: ${explained.join(', ')}\n` : '')
.write();

explain:
needs: plan
if: needs.plan.outputs.has-reports == 'true'
name: "Explain ${{ matrix.python }}:${{ matrix.mode }}"
runs-on: ${{ matrix.runner }}
# Explaining constraints-source-providers took up to 115 minutes in the canary (Oct 2026).
timeout-minutes: 150
strategy:
fail-fast: false
max-parallel: 3
matrix: ${{ fromJSON(needs.plan.outputs.matrix) }}
env:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_USERNAME: ${{ github.actor }}
VERBOSE: "false"
USE_UV: ${{ matrix.use-uv }}
steps:
- name: "Checkout the tested revision"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.plan.outputs.source-sha }}
persist-credentials: false
# The disk is prepared as in the image build jobs: the loaded image and uv resolutions outgrow /,
# and the runner scripts come from the workflow's own version, which knows RunsOn's disks.
- name: "Checkout the workflow version for its actions and runner scripts"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
path: workflow-version
sparse-checkout: |
.github/actions
scripts/ci
- name: "Free up disk space"
shell: bash
run: ./scripts/tools/free_up_disk_space.sh
- name: "Make /mnt writeable"
run: ./workflow-version/scripts/ci/make_mnt_writeable.sh
- name: "Move docker to /mnt"
run: ./workflow-version/scripts/ci/move_docker_to_mnt.sh
- name: "Install Breeze"
uses: ./.github/actions/breeze
- name: "Download the original CI image"
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
artifact-ids: ${{ matrix.image-id }}
run-id: ${{ needs.plan.outputs.source-run-id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
merge-multiple: true
path: /mnt
if: matrix.image-source == 'artifact'
- name: "Download the digest of the original CI image"
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
artifact-ids: ${{ matrix.image-id }}
run-id: ${{ needs.plan.outputs.source-run-id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
merge-multiple: true
path: ${{ runner.temp }}/ci-image-digest
if: matrix.image-source == 's3'
# The tarball is the one the canary stashed, addressed by its digest as in
# .github/actions/stash-restore, so a later build on `main` cannot replace it.
- name: "Restore the original CI image from the RunsOn cache"
shell: bash
run: |
set -euo pipefail
digest="$(cat "${RUNNER_TEMP}/ci-image-digest/sha256")"
if [[ ! "${digest}" =~ ^[0-9a-f]{64}$ ]]; then
echo "::error::Malformed digest of the original CI image"
exit 1
fi
export AWS_DEFAULT_REGION="${RUNS_ON_AWS_REGION:?not running on a RunsOn runner}"
bucket="${RUNS_ON_S3_BUCKET_CACHE:?not running on a RunsOn runner}"
object="s3://${bucket}/cache/airflow-stash/sha256/${digest}.tar"
payload="${RUNNER_TEMP}/ci-image-stash.tar"
aws s3 cp --only-show-errors "${object}" "${payload}"
echo "${digest} ${payload}" | sha256sum --check --status
tar -tPf "${payload}" | while IFS= read -r entry; do
if [[ "${entry}" != /mnt/* || "${entry}" == */../* || "${entry}" == */.. ]]; then
echo "::error::The original CI image stash holds ${entry} outside /mnt"
exit 1
fi
done
tar -xPf "${payload}"
rm -f "${payload}"
if: matrix.image-source == 's3'
- name: "Load the original CI image"
env:
PYTHON_VERSION: ${{ matrix.python }}
run: breeze ci-image load --platform linux/amd64 --python "${PYTHON_VERSION}" --image-file-dir /mnt
- name: "Download the constraints used by the quick summary"
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
artifact-ids: ${{ matrix.inputs-id }}
run-id: ${{ needs.plan.outputs.source-run-id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
merge-multiple: true
path: files/dependency-report
- name: "Explain outdated dependencies"
shell: bash
env:
PYTHON_VERSION: ${{ matrix.python }}
CONSTRAINTS_MODE: ${{ matrix.mode }}
SOURCE_RUN_ID: ${{ needs.plan.outputs.source-run-id }}
SOURCE_SHA: ${{ needs.plan.outputs.source-sha }}
run: |
{
echo "## Dependency upgrade explanations"
echo "Source canary run: ${SOURCE_RUN_ID}; source revision: ${SOURCE_SHA}"
echo "Constraints commit: $(cat files/dependency-report/constraints-commit.txt)"
echo "Python: ${PYTHON_VERSION}; mode: ${CONSTRAINTS_MODE}"
echo "The constraints are the constraints branch the canary's summary checked, not ones built"
echo "from the source revision: dependency bounds changed between the two can show up below as"
echo "conflicts or as stale constraints."
} >> "${GITHUB_STEP_SUMMARY}"
breeze release-management constraints-version-check \
--python "${PYTHON_VERSION}" --airflow-constraints-mode "${CONSTRAINTS_MODE}" \
--constraints-file files/dependency-report/constraints.txt --explain-why \
2>&1 | tee files/dependency-report/explanations.txt
- name: "Save diagnostic output, including partial output on failure"
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: dependency-explanations-${{ matrix.python }}-${{ matrix.mode }}
path: files/dependency-report/
if-no-files-found: ignore
retention-days: 7
overwrite: true
29 changes: 25 additions & 4 deletions .github/workflows/finalize-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,16 +153,37 @@ jobs:
python: ${{ matrix.python-version }}
use-uv: ${{ inputs.use-uv }}
make-mnt-writeable-and-cleanup: true
# Scheduled and dispatched canaries on main leave the explanations to dependency-upgrade-report.yml
# and save the constraints they checked for it. Saving them never fails this step.
- name: "Deps: ${{ matrix.python-version }}:${{ matrix.constraints-mode }}"
shell: bash
run: >
breeze release-management constraints-version-check
--python "${MATRIX_PYTHON_VERSION}"
--airflow-constraints-mode "${MATRIX_CONSTRAINTS_MODE}" --explain-why
run: |
REPORT_ARGS=(--explain-why)
if [[ "${INDEPENDENT_REPORT}" == "true" ]]; then
REPORT_ARGS=(--no-explain-why --save-constraints-to files/dependency-report)
fi
breeze release-management constraints-version-check \
--python "${MATRIX_PYTHON_VERSION}" \
--airflow-constraints-mode "${MATRIX_CONSTRAINTS_MODE}" "${REPORT_ARGS[@]}"
env:
MATRIX_PYTHON_VERSION: "${{ matrix.python-version }}"
MATRIX_CONSTRAINTS_MODE: "${{ matrix.constraints-mode }}"
INDEPENDENT_REPORT: >-
${{ github.ref == 'refs/heads/main' &&
(github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') }}
VERBOSE: "false"
- name: "Upload dependency report inputs"
if: >-
!cancelled() && hashFiles('files/dependency-report/constraints-commit.txt') != ''
continue-on-error: true
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: "dependency-report-inputs-${{ matrix.python-version }}-${{ matrix.constraints-mode }}\
-${{ inputs.use-uv }}-${{ github.run_attempt }}"
path: files/dependency-report/
if-no-files-found: error
retention-days: 2
overwrite: true

push-buildx-cache-to-github-registry:
name: Push Regular Image Cache ${{ inputs.platform }}
Expand Down
45 changes: 45 additions & 0 deletions dev/breeze/doc/ci/05_workflows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,6 +563,51 @@ unprivileged, which constrains what can go on it:
When adding a job, reach for `ubuntu-slim` if it only shuffles metadata around, and
`ubuntu-22.04` otherwise.

### Dependency upgrade diagnostics

Scheduled and manually dispatched AMD canaries on `main` keep the dependency freshness
summary in `finalize-tests.yml`, but run it with `--no-explain-why`. Tests, constraint
generation and validation, and image publication still run in the canary. Release-branch
and other run types retain their existing explanation behavior. The summary saves the
constraints it checked, and the constraints-branch commit they came from, with
`--save-constraints-to`; a failure to save them only warns and never fails the canary.

After the canary completes, `dependency-upgrade-report.yml` runs the detailed explanations
independently, from the inputs the canary saved; a canary with failed jobs saves none. It checks out
the original source SHA and downloads the original image and saved constraints by artifact
ID from that run. It never substitutes the latest branch image or constraints: an entry whose
saved constraints or image are missing or expired is skipped with a warning and listed in the
summary, and the others are still explained. The image comes from the CI image stash, which
keeps only the newest image per Python version and branch, so a later build on `main` can
replace it within hours. A canary on RunsOn keeps the image in RunsOn's S3 cache and stashes
only its digest; the report job then runs on a RunsOn runner and downloads exactly that tarball
by its digest. Re-running the canary triggers the workflow again; inputs that an earlier attempt
already saved are not explained again.

The saved constraints are the constraints branch as the summary saw it, not constraints
generated from the source SHA, so dependency bounds that changed between the two can appear
as conflicts or as stale constraints. The report summary shows both commits. PyPI release
metadata is fetched when the report executes, so the available upgrade targets can change
between the summary and the explanations.

The workflow has read-only repository permissions. It runs on `workflow_run`, in the context
of the default branch, so it could write that branch's GitHub Actions cache: what keeps the
cache untouched is that no step saves one. Do not add caching (for example `actions/cache` or
the cache option of a `setup-*` action) to this workflow. Reports use the saved image artifact.

The report runs at most three jobs concurrently. Each job has a 150-minute limit and saves
its output, including partial output on failure. A failed or timed-out report is incomplete;
it does not invalidate the canary's test result. Find diagnostics in the **Dependency
upgrade report** workflow, whose summary links to the source canary. To repeat diagnostics,
use **Re-run jobs** on that report before its two-day input retention expires (or run
`gh run rerun REPORT_RUN_ID`). Reruns keep the original source canary ID and attempt. A rerun
fails when inputs were saved but none can be explained any more, rather than using a newer
image or constraints; a report whose canary saved no inputs posts the same notice every time.

`breeze release-management constraints-version-check --constraints-file PATH` can also
analyze a saved constraints file locally. Detailed explanations reuse one baseline resolution
and skip the pinned resolution when that baseline already selects the target version.

### Spot interruptions of self-hosted runners

Runs of `apache/airflow` committers and the canary run on self-hosted
Expand Down
Loading
Loading