Repository navigation
Serialise the workflows that publish shared state - #75996
Merged
Merged
Conversation
An incremental registry build downloads the bucket's providers.json and modules.json, merges its own providers in and syncs them back. Docs publishes call it, release managers fire subset and fix-up publishes, and the backfill writes the same bucket, so two of them overlapping silently drop whichever finished first from the registry index. Docs publishes to one bucket and the two constraints publishers had the same exposure in smaller form. Provider and Airflow docs land in different paths, so those two kinds of docs sync keep running side by side. The packages-metadata.json they both rewrite from a scan of the whole bucket moves to its own job, serialised per bucket, using new --skip-packages-metadata and --packages-metadata-only flags of publish-docs-to-s3. The Docker Hub release grouped on the bare version string, which any other workflow using a version as its group would collide with. Generated-by: Claude Opus 5
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
vatsrahul1001
as code owners
October 29, 2026 22:44
Contributor
|
The re-run of the jobs stopped by EC2 Spot interruptions finished: attempt 4 failed. |
Member
Author
Indeed - it looks like a bug in a hardened image configuration - from what I saw, while they are reacting quickly to realeas security fixes, there are some cases (it already happened once) where some hardened packages conflict with regular packages from debian. I will take a close look. |
Member
Author
|
Fix is coming. |
Member
Author
|
In a separate PR. |
Contributor
|
Hi maintainer, this PR was merged without a milestone set.
|
Contributor
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
potiuk
added a commit
that referenced
this pull request
Oct 30, 2026
An incremental registry build downloads the bucket's providers.json and modules.json, merges its own providers in and syncs them back. Docs publishes call it, release managers fire subset and fix-up publishes, and the backfill writes the same bucket, so two of them overlapping silently drop whichever finished first from the registry index. Docs publishes to one bucket and the two constraints publishers had the same exposure in smaller form. Provider and Airflow docs land in different paths, so those two kinds of docs sync keep running side by side. The packages-metadata.json they both rewrite from a scan of the whole bucket moves to its own job, serialised per bucket, using new --skip-packages-metadata and --packages-metadata-only flags of publish-docs-to-s3. The Docker Hub release grouped on the bare version string, which any other workflow using a version as its group would collide with. (cherry picked from commit c6a077b) Co-authored-by: Jarek Potiuk <jarek@potiuk.com> Generated-by: Claude Opus 5
potiuk
added a commit
that referenced
this pull request
Oct 30, 2026
#76022) An incremental registry build downloads the bucket's providers.json and modules.json, merges its own providers in and syncs them back. Docs publishes call it, release managers fire subset and fix-up publishes, and the backfill writes the same bucket, so two of them overlapping silently drop whichever finished first from the registry index. Docs publishes to one bucket and the two constraints publishers had the same exposure in smaller form. Provider and Airflow docs land in different paths, so those two kinds of docs sync keep running side by side. The packages-metadata.json they both rewrite from a scan of the whole bucket moves to its own job, serialised per bucket, using new --skip-packages-metadata and --packages-metadata-only flags of publish-docs-to-s3. The Docker Hub release grouped on the bare version string, which any other workflow using a version as its group would collide with. (cherry picked from commit c6a077b) Generated-by: Claude Opus 5 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
vatsrahul1001
pushed a commit
that referenced
this pull request
Oct 30, 2026
#76022) An incremental registry build downloads the bucket's providers.json and modules.json, merges its own providers in and syncs them back. Docs publishes call it, release managers fire subset and fix-up publishes, and the backfill writes the same bucket, so two of them overlapping silently drop whichever finished first from the registry index. Docs publishes to one bucket and the two constraints publishers had the same exposure in smaller form. Provider and Airflow docs land in different paths, so those two kinds of docs sync keep running side by side. The packages-metadata.json they both rewrite from a scan of the whole bucket moves to its own job, serialised per bucket, using new --skip-packages-metadata and --packages-metadata-only flags of publish-docs-to-s3. The Docker Hub release grouped on the bare version string, which any other workflow using a version as its group would collide with. (cherry picked from commit c6a077b) Generated-by: Claude Opus 5 Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several workflows read, modify and write back state that other runs share. When two of them overlap, one run's result is silently lost.
providers.jsonandmodules.json, merges in its own providers and syncs the files back. Docs publishes trigger it, release managers fire subset and fix-up publishes, and the backfill writes to the same bucket. Two overlapping runs drop whichever one finished first from the index. The registry publish job and the backfill'spublish-versionsjob now shareregistry-publish-<destination>. The group is set on the job becauseregistry-build.ymlis also called frompublish-docs-to-s3.yml.publish-docs-<destination>-<providers|airflow>).manifest/packages-metadata.jsonis different: it is rebuilt from a scan of the whole bucket, so both kinds share it. Rebuilding it now runs in a separate job,update-packages-metadata, with one group per bucket. That job runs only when the sync step succeeded, so a schema-publish failure after a complete sync still updates the metadata, and a partial sync is never advertised. The newpublish-docs-to-s3flags--skip-packages-metadataand--packages-metadata-onlymake this split possible.release-constraints.yml,update-constraints-on-push.ymlandupdate-constraints-on-push-stable.ymlpush to the sameconstraints-X-Ybranch. Their publish jobs now shareconstraints-publish-<branch>, so a release tag push and a refresh no longer race into a non-fast-forward. The stable workflow runs from thev*-stablebranches, so it takes effect there only after a backport.release-dockerhub-image-prefix.All new groups use
cancel-in-progress: false.Known limitations
airflow. For example,include-docsset toallor toapache-airflow amazongets theairflowgroup, but it also writes provider paths. Such a run can still overlap a provider-only publish of the same provider. Covering this would mean splitting the publish into separately grouped provider and non-provider jobs. Mixed publishes are rare manual runs, so this PR does not do that.Checks run
uv run --project dev/breeze pytest dev/breeze/tests/test_publish_docs_to_s3.py dev/breeze/tests/test_release_management_commands.py -k publish_docs_to_s3: passed. The new tests cover skipping the metadata during a sync, the metadata-only mode, and option validation.breeze setup regenerate-command-images --check-onlyandbreeze setup check-all-params-in-groups: passed.prek run --from-ref main --stage pre-commit: passed.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines
🤖 Generated with Claude Code