Repository navigation

Sync v3-4-stable with v3-4-test to release 3.4.0rc1 #75755

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

vatsrahul1001 merged 103 commits into v3-4-stable from changes-3.4.0rc1

Oct 28, 2026

.agents/skills/airflow-translations/locales/zh-TW.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -29,8 +29,13 @@ Prioritize terms commonly used in the Taiwan tech ecosystem. @@
     ## Plural Forms
-    Traditional Chinese **does not** distinguish between singular and plural forms.
-    Use the **same translation** for both `_one` and `_other` suffixes:
+    Traditional Chinese **does not** distinguish between singular and plural forms,
+    so both suffixes use the same noun. i18next currently never selects `_one`
+    for `zh-TW` (the locale has a single plural category, `other`), but the key
+    must still exist for translation completeness.
+    Some keys are plain nouns rather than counts — use the same translation for
+    both suffixes:
     **English source:**
@@ Expand All @@
     "dagRun_other": "Dag 執行"
     ```
+    When the English `_one` does carry a count, write it as a literal `1` instead
+    of interpolating `{{count, number}}`; keep the placeholder in `_other`:
+    **English source:**
+    ```json
+    "taskCount_one": "{{count, number}} Task",
+    "taskCount_other": "{{count, number}} Tasks"
+    ```
+    **Correct** — literal `1` in `_one`:
+    ```json
+    "taskCount_one": "1 個任務",
+    "taskCount_other": "{{count, number}} 個任務"
+    ```
+    **Incorrect** — interpolating a count that can only be 1:
+    ```json
+    "taskCount_one": "{{count, number}} 個任務",
+    "taskCount_other": "{{count, number}} 個任務"
+    ```
     ## Spacing Rules
     Insert a **half-width space** between Chinese characters and adjacent
@@ Expand Down @@

.asf.yaml

-Original file line number
+Diff line change
@@ Expand Up / @@ -72,8 +72,11 @@ github: @@
         rebase: false
       pull_requests:
-        # allow auto-merge
-        allow_auto_merge: false
+        # Auto-merge waits for the required "CI passed" check enforced by the "Require CI on main" and
+        # "Require CI on release branches" rulesets. Without a required check it would merge as soon as
+        # the PR is approved, before CI finishes. The setting is repository-wide, so every branch that
+        # PRs target needs one of those rulesets.
+        allow_auto_merge: true
         # enable updating head branches of pull requests
         allow_update_branch: true
         # auto-delete head branches after being merged
@@ Expand Down Expand Up / @@ -104,7 +107,64 @@ github: @@
           required_conversation_resolution: true
           required_signatures: false
+      # The two CI rulesets use the raw Rulesets API payload, because the convenience syntax can only name
+      # teams as bypass actors and GitHub refuses secret teams such as `airflow-committers`. Instead the
+      # Write repository role (actor_id 4) bypasses them: on this repository only committers hold it,
+      # granted by ASF Infra from the `airflow` LDAP group, so it needs no update when committers change.
+      # Committers can still merge a PR whose CI is red for an unrelated reason, by bypassing the rule.
+      #
+      # The required checks are not pinned to the GitHub Actions app: asfyaml passes raw values to the
+      # API as strings except for a fixed set of keys, and a string `integration_id` is rejected.
       rulesets:
+        - name: "Require CI on main"
+          target: branch
+          enforcement: active
+          conditions:
+            ref_name:
+              include:
+                - "refs/heads/main"
+              exclude: []
+          bypass_actors:
+            - actor_id: 4
+              actor_type: RepositoryRole
+              bypass_mode: always
+          rules:
+            - type: deletion
+            - type: non_fast_forward
+            - type: required_status_checks
+              parameters:
+                strict_required_status_checks_policy: false
+                required_status_checks:
+                  - context: "CI passed"
+        # The branches `ci-amd.yml` runs pull request CI for, apart from `main`. A release branch gets
+        # the "CI passed" job only when it is backported there; until then its PRs need a committer to
+        # bypass the rule.
+        - name: "Require CI on release branches"
+          target: branch
+          enforcement: active
+          conditions:
+            ref_name:
+              include:
+                - "refs/heads/v*-test"
+                - "refs/heads/v*-stable"
+                - "refs/heads/providers-*/v*"
+                - "refs/heads/chart/v*-test"
+                - "refs/heads/chart/v*-stable"
+                - "refs/heads/airflow-ctl/v*-test"
+                - "refs/heads/airflow-ctl/v*-stable"
+              exclude: []
+          bypass_actors:
+            - actor_id: 4
+              actor_type: RepositoryRole
+              bypass_mode: always
+          rules:
+            - type: deletion
+            - type: non_fast_forward
+            - type: required_status_checks
+              parameters:
+                strict_required_status_checks_policy: false
+                required_status_checks:
+                  - context: "CI passed"
         - name: "Branch protection"
           type: branch
           branches:
@@ Expand Down @@

.github/AGENTS.md

-Original file line number
+Diff line change
@@ -0,0 +1,38 @@
+    <!-- SPDX-License-Identifier: Apache-2.0
+         https://www.apache.org/licenses/LICENSE-2.0 -->
+    <!-- START doctoc generated TOC please keep comment here to allow auto update -->
+    <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+    **Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
+    - [The `.github` folder — Agent Instructions](#the-github-folder--agent-instructions)
+      - [Keep the CI-minutes estimates in sync with the workflows](#keep-the-ci-minutes-estimates-in-sync-with-the-workflows)
+    <!-- END doctoc generated TOC please keep comment here to allow auto update -->
+    # The `.github` folder — Agent Instructions
+    ## Keep the CI-minutes estimates in sync with the workflows
+    `breeze ci approve-workflow-runs` (run by `workflows/auto-approve-workflow-runs.yml`) decides whether
+    to approve an external contributor's CI runs from an estimate of the job-minutes they take. The
+    numbers live in `dev/breeze/src/airflow_breeze/utils/ci_minutes_estimate.py` and are medians
+    measured on recent `Tests (AMD)` PR runs on public runners.
+    After you change the CI workflows in a way that adds, removes, splits or resizes jobs (new test
+    groups, a changed matrix, new composite workflows, a move between runner types), refresh the
+    estimates once the change has run on enough PRs, and in any case about once a month:
+. Run `uv run dev/calibrate_ci_minutes_estimates.py --since <date>`
+       from the repository root with an authenticated `gh` CLI. Pick a date that covers at least a
+       week of runs after the workflow change.
+. Copy the medians and fits it prints into the constants of `ci_minutes_estimate.py`, and update
+       the "Last calibrated" line next to them.
+. When you add a new job group to the workflows, add a matching group to both
+       `ci_minutes_estimate.py` (with the selective-check output that turns it on) and the
+       `JOB_GROUPS` patterns of the calibration script. Unmatched jobs fall into the base group.
+. Run `uv run --project dev/breeze pytest dev/breeze/tests/test_ci_minutes_estimate.py
+       dev/breeze/tests/test_workflow_run_approval.py -xvs`.
+    The estimates cover public runners only. Self-hosted (RunsOn) runners need their own set once there
+    is enough data from them.

.github/actions/breeze/action.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -29,6 +29,8 @@ outputs: @@
     runs:
       using: "composite"
       steps:
+        - name: "RunsOn cost summary"
+          uses: ./.github/actions/runs-on-cost-summary
         - name: "Setup python"
           uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
           with:
@@ Expand Down @@

.github/actions/prepare_breeze_and_image/action.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -82,12 +82,13 @@ runs: @@
         - name: >
             Restore ${{ inputs.image-type }} docker image ${{ inputs.platform }}:${{ inputs.python }}
             ${{ inputs.image-stash-ref != '' && format('built for ref {0}', inputs.image-stash-ref) || '' }}
-          # yamllint disable-line rule:line-length
-          uses: apache/infrastructure-actions/stash/restore@61dcea11f19e2bbe1263f14d72235e8da17d3ad0  # restore/v1.0.0
+          uses: ./.github/actions/stash-restore
           with:
             key: "${{ inputs.image-type }}-image-save-v3-${{ inputs.platform }}-${{ inputs.python }}\
               ${{ inputs.image-stash-ref != '' && format('-{0}', inputs.image-stash-ref) || '' }}"
             path: "/mnt/"
+            # Must stay true: runs on RunsOn (canaries and release-branch pushes) save these image
+            # stashes as S3 digests only.
             only-current-branch: 'true'
             fail-on-download: 'true'
         - name: "Load ${{ inputs.image-type }} image ${{ inputs.platform }}:${{ inputs.python }}"
@@ Expand Down @@

.github/actions/prepare_single_ci_image/action.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -36,11 +36,12 @@ runs: @@
       using: "composite"
       steps:
         - name: "Restore CI docker images ${{ inputs.platform }}:${{ inputs.python }}"
-          # yamllint disable-line rule:line-length
-          uses: apache/infrastructure-actions/stash/restore@61dcea11f19e2bbe1263f14d72235e8da17d3ad0  # restore/v1.0.0
+          uses: ./.github/actions/stash-restore
           with:
             key: ci-image-save-v3-${{ inputs.platform }}-${{ inputs.python }}
             path: "/mnt/"
+            # Must stay true: runs on RunsOn (canaries and release-branch pushes) save these image
+            # stashes as S3 digests only.
             only-current-branch: 'true'
             fail-on-download: 'true'
           if: contains(inputs.python-versions-list-as-string, inputs.python)
@@ Expand Down @@

.github/actions/runs-on-cost-summary/action.yml

-Original file line number
+Diff line change
@@ -0,0 +1,30 @@
+    # Licensed to the Apache Software Foundation (ASF) under one
+    # or more contributor license agreements.  See the NOTICE file
+    # distributed with this work for additional information
+    # regarding copyright ownership.  The ASF licenses this file
+    # to you under the Apache License, Version 2.0 (the
+    # "License"); you may not use this file except in compliance
+    # with the License.  You may obtain a copy of the License at
+    #
+    #   http://www.apache.org/licenses/LICENSE-2.0
+    #
+    # Unless required by applicable law or agreed to in writing,
+    # software distributed under the License is distributed on an
+    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    # KIND, either express or implied.  See the License for the
+    # specific language governing permissions and limitations
+    # under the License.
+    #
+    ---
+    # Puts the job's EC2 cost next to the GitHub-hosted price in the job summary. Used by every job
+    # that can run on RunsOn: through the breeze action, or directly by jobs that do not install Breeze.
+    name: 'RunsOn cost summary'
+    description: 'Add the EC2 cost of a RunsOn job to its summary; a no-op elsewhere.'
+    runs:
+      using: 'composite'
+      steps:
+        - name: "RunsOn cost summary"
+          if: runner.environment == 'self-hosted'
+          uses: runs-on/action@17b59092eb37ebfdbb034460892887c09100eaa3  # v2.5.0
+          with:
+            show_costs: summary

.github/actions/stash-restore/action.yml

-Original file line number
+Diff line change
@@ -0,0 +1,137 @@
+    # Licensed to the Apache Software Foundation (ASF) under one
+    # or more contributor license agreements.  See the NOTICE file
+    # distributed with this work for additional information
+    # regarding copyright ownership.  The ASF licenses this file
+    # to you under the Apache License, Version 2.0 (the
+    # "License"); you may not use this file except in compliance
+    # with the License.  You may obtain a copy of the License at
+    #
+    #   http://www.apache.org/licenses/LICENSE-2.0
+    #
+    # Unless required by applicable law or agreed to in writing,
+    # software distributed under the License is distributed on an
+    # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    # KIND, either express or implied.  See the License for the
+    # specific language governing permissions and limitations
+    # under the License.
+    #
+    ---
+    # apache/infrastructure-actions/stash/restore on GitHub-hosted runners. On RunsOn (the only
+    # self-hosted runners) multi-GB image tarballs stay in the RunsOn stack's in-region S3 cache
+    # bucket instead of leaving AWS as GitHub artifacts.
+    #
+    # Every RunsOn job can write anywhere in that bucket, so the bucket is not trusted: objects are
+    # named by their sha256, and the digest itself travels as a small ASF stash artifact, which
+    # GitHub scopes to the run / branch exactly like a regular stash. A blob that does not match the
+    # digest the ASF restore found for this run, branch and repository is never extracted. The bucket's
+    # lifecycle rule (S3CacheExpirationInDays) must outlive the longest retention-days of a digest
+    # (6 days today), or a digest hit finds no blob.
+    name: 'Restore stash'
+    description: 'Restore a stash from GitHub artifacts, or from S3 on RunsOn runners.'
+    inputs:
+      key:
+        description: 'Stash key.'
+        required: true
+      path:
+        description: >
+          Directory to restore into (GitHub-hosted only; the S3 tarball restores entries to the
+          paths they were saved from, and the restore then fails if any of them is outside this directory).
+        required: false
+        default: ${{ github.workspace }}
+      only-current-branch:
+        description: 'Do not fall back to the base branch stash.'
+        required: false
+        default: 'false'
+      fail-on-download:
+        description: 'Fail when a stash was found but could not be restored.'
+        required: false
+        default: 'false'
+    outputs:
+      stash-hit:
+        description: "'true' if a stash was restored, 'false' otherwise."
+        value: ${{ steps.s3.outputs.stash-hit || steps.artifact.outputs.stash-hit }}
+    runs:
+      using: 'composite'
+      steps:
+        - name: "Restore stash from GitHub artifacts"
+          id: artifact
+          if: runner.environment != 'self-hosted'
+          # yamllint disable-line rule:line-length
+          uses: apache/infrastructure-actions/stash/restore@c968777141a2afc98d1a09c9f6979455bfebaae3  # restore/v1.0.1
+          with:
+            key: ${{ inputs.key }}
+            path: ${{ inputs.path }}
+            only-current-branch: ${{ inputs.only-current-branch }}
+            fail-on-download: ${{ inputs.fail-on-download }}
+        - name: "Restore stash digest from GitHub artifacts"
+          id: digest
+          if: runner.environment == 'self-hosted'
+          # yamllint disable-line rule:line-length
+          uses: apache/infrastructure-actions/stash/restore@c968777141a2afc98d1a09c9f6979455bfebaae3  # restore/v1.0.1
+          with:
+            key: ${{ inputs.key }}-s3
+            path: ${{ runner.temp }}/stash-digest-restore/${{ inputs.key }}
+            only-current-branch: ${{ inputs.only-current-branch }}
+            fail-on-download: ${{ inputs.fail-on-download }}
+        - name: "Restore stash from S3"
+          id: s3
+          if: runner.environment == 'self-hosted'
+          shell: bash
+          env:
+            STASH_KEY: ${{ inputs.key }}
+            STASH_PATH: ${{ inputs.path }}
+            DIGEST_HIT: ${{ steps.digest.outputs.stash-hit }}
+            DIGEST_FILE: ${{ runner.temp }}/stash-digest-restore/${{ inputs.key }}/sha256
+            FAIL_ON_DOWNLOAD: ${{ inputs.fail-on-download }}
+          run: |
+            set -uo pipefail
+            payload="${RUNNER_TEMP}/stash-restore-$$.tar"
+            entries="${RUNNER_TEMP}/stash-restore-$$.entries"
+            miss() {
+              rm -f "${payload}" "${entries}"
+              echo "stash-hit=false" >> "${GITHUB_OUTPUT}"
+              if [[ "${FAIL_ON_DOWNLOAD}" == "true" && "${DIGEST_HIT}" == "true" ]]; then
+                echo "::error::$1"
+                exit 1
+              fi
+              echo "$1"
+              exit 0
+            }
+            if [[ "${DIGEST_HIT}" != "true" ]]; then
+              miss "No stash '${STASH_KEY}'"
+            fi
+            digest="$(cat "${DIGEST_FILE}")"
+            if [[ ! "${digest}" =~ ^[0-9a-f]{64}$ ]]; then
+              miss "Malformed digest for stash '${STASH_KEY}'"
+            fi
+            bucket="${RUNS_ON_S3_BUCKET_CACHE:?not running on a RunsOn runner}"
+            export AWS_DEFAULT_REGION="${RUNS_ON_AWS_REGION:?not running on a RunsOn runner}"
+            object="s3://${bucket}/cache/airflow-stash/sha256/${digest}.tar"
+            if ! aws s3 cp --only-show-errors "${object}" "${payload}"; then
+              miss "Could not download ${object} for stash '${STASH_KEY}'"
+            fi
+            if ! echo "${digest}  ${payload}" | sha256sum --check --status; then
+              miss "${object} does not match the digest recorded for stash '${STASH_KEY}'"
+            fi
+            # GNU tar detects the compression stash-save chose.
+            if ! tar -xPvf "${payload}" > "${entries}"; then
+              miss "Could not extract ${object} for stash '${STASH_KEY}'"
+            fi
+            rm -f "${payload}"
+            # The S3 tarball ignores `path`: fail loudly if a caller saves and restores different
+            # directories, which would work on GitHub-hosted runners only.
+            restore_dir="$(realpath -ms "${STASH_PATH}")"
+            restore_dir="${restore_dir%/}"
+            outside=()
+            while IFS= read -r entry; do
+              [[ "${entry}" == /* ]] || entry="${GITHUB_WORKSPACE}/${entry}"
+              [[ "${entry}" == "${restore_dir}/"* ]] || outside+=("${entry}")
+            done < "${entries}"
+            rm -f "${entries}"
+            if [[ ${#outside[@]} -gt 0 ]]; then
+              printf '%s\n' "${outside[@]}"
+              echo "::error::Stash '${STASH_KEY}' restored the entries above outside of ${restore_dir}/"
+              exit 1
+            fi
+            echo "stash-hit=true" >> "${GITHUB_OUTPUT}"
+            echo "Restored stash '${STASH_KEY}' from ${object}"

Navigation Menu

Repository navigation

Sync v3-4-stable with v3-4-test to release 3.4.0rc1 #75755

Uh oh!

Diff view

Diff view

Uh oh!

There are no files selected for viewing

Uh oh!

Uh oh!