Fix registry backfill with per-provider versions and Docker extraction#65223
Conversation
potiuk
left a comment
There was a problem hiding this comment.
My comments are not blocking - but I have a feeling things can be quite simplified with leveraging uv
Chain both extraction scripts in a single uv run invocation to avoid creating two ephemeral environments per version.
Backport failed to create: v3-2-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker cda493d v3-2-testThis should apply the commit to the v3-2-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
The registry backfill workflow had two problems preventing it from backfilling intermediate provider versions:
Flat version list applied to all providers: Input like
providers="amazon google" versions="9.24.0 21.0.0"tried all versions against all providers, failing because amazon doesn't have 21.0.0 and google doesn't have 9.24.0.Missing system dependencies: Extraction ran
uv run --withon a bare GitHub runner, which lacks C libraries (krb5-dev, libxml2-dev) needed by providers like amazon and google.Changes
Workflow (
registry-backfill.yml):providers+versionsinputs withprovider-versionsacceptingprovider/versionpairs (e.g.amazon/9.24.0 google/21.0.0 amazon/9.23.0)build-ci-imagejob andprepare_breeze_and_imagestep (same pattern asregistry-build.yml)Breeze command (
registry_commands.py):_backfill_docker()that runs extraction inside the Breeze CI container viapip install+execute_command_in_shell_backfill_uv()as--no-dockerfallback--no-dockerfor faster iterationExample usage
This creates parallel jobs: one for amazon (9.24.0), one for google (21.0.0), one for celery (3.17.2). Multiple versions per provider are grouped into a single job.