diff --git a/.agents/skills/prepare-providers-documentation/SKILL.md b/.agents/skills/prepare-providers-documentation/SKILL.md index e03021939ae19..79f85d7443c87 100644 --- a/.agents/skills/prepare-providers-documentation/SKILL.md +++ b/.agents/skills/prepare-providers-documentation/SKILL.md @@ -176,6 +176,25 @@ How to read it: > back as `needs_llm`. The rules live in `classify_change_deterministically` > (`dev/breeze/src/airflow_breeze/prepare_providers/provider_documentation.py`). +The range the classifier reports is not fixed for the whole run — it widens once +a provider actually gets a version bump: + +> [!WARNING] +> **A version bump supersedes the provider's doc-only marker — re-run discovery +> after Phase 4a.** When `docs/.latest-doc-only-change.txt` is present the +> classifier starts the range at that commit, so anything older (earlier doc-only +> changes, tooling churn) is hidden for as long as the provider has no pending +> release. That is correct while nothing ships — but the moment the provider gets +> a real version bump the marker no longer applies, and those commits are part of +> what the release publishes. +> +> Re-run `classify-provider-changes` **after** bumping versions and fold in every +> commit that newly appears; otherwise the release silently drops doc-only +> entries that were only ever suppressed by the marker. The tell is a provider +> whose commit count grows between two runs without any new merges. Observed in +> the 2026-08-01 wave: `cohere` gained #69649, `dbt.cloud`/`presto`/`trino` gained +> #69478, and `papermill` gained #68322 — all invisible in the first pass. + Then regenerate the auto-generated build files (this does **no** classification, so nothing random is produced): @@ -545,6 +564,33 @@ Rules: indent, double backticks). - Subjects must be the original commit subject with backticks replaced by single quotes (matches `message_without_backticks`). Don't paraphrase. +- **Exception — rewrite subjects written in project-internal language.** The + don't-paraphrase rule keeps you honest about *what* shipped; it does not + oblige you to publish a subject the reader cannot act on. The changelog is + read by users, not by us. Rewrite the entry — keeping every `(#NNNN)` — when + the subject is only meaningful inside the project: + - **Spell out internal abbreviations.** `KPO`, `DFP`, `TI`, `RTIF`, `OL` and + friends are our shorthand. Write ``Add '--min-completed-minutes' to + 'cleanup-pods' to prevent KubernetesPodOperator race condition (#70595)``, + not ``… to prevent KPO race condition (#70595)``. + - **Name what a grouped dependency bump actually changed.** Dependabot group + subjects such as ``Bump the fab-ui-package-updates group across 1 directory + with 3 updates (#70604)`` tell the reader nothing. Read the PR diff and list + the packages with their versions: ``Bump prettier to 3.9.6, stylelint to + 17.14.1, webpack to 5.109.0 (#70604)``. A single-package bump still needs + its target version: ``Bump eslint to 10.8.0 (#70697)``. + - **Describe the user-visible effect, not the internal mechanic.** ``Remove + noqa:S101 from production code (#70378)`` names a lint directive; the reader + wants ``Mark asserts under 'TYPE_CHECKING' in 'DocumentLoaderOperator' + (#70378)``. + - **Drop our test vocabulary.** A system test is just an example Dag to the + user, so ``Remove hard-coded deferrable crawler run from example_glue system + test (#70206)`` becomes ``Remove hard-coded deferrable crawler run from + example_glue (#70206)``. + + Rewrite the *wording*, never the *claim* — do not describe behaviour that did + not ship. When a subject is too vague to rewrite honestly, read the PR diff + before writing the entry. - **Exception — collapse within-wave "add then rename/rework" chains into one net entry.** When several pending commits are steps toward *one* net change — a feature added in one PR and renamed or reworked in a later PR, both since @@ -672,6 +718,20 @@ provider-by-provider: heading and the first version), notify the release manager immediately — it likely means a breaking-change or min-version note was accidentally written at the wrong indentation level or position. + + Two variants to check for, both seen in the 2026-08-01 wave: + - **Already misplaced on the base branch.** A contributor adds a note for + their own change directly under the `Changelog` header instead of inside + the version section it belongs to, so it renders page-wide. Move it into + the section that ships the change (`google` #70869). + - **Pushed out of place by your own prepend.** When a note was sitting above + the first version header and you prepend a new section, the note ends up + *between* your new excluded block and the previous version — still wrong, + but no longer above the first header, so a scan that only looks at the top + of the file misses it (`openai` #69506). Check **inside every section you + touched** that no `.. note::` appears after the + `.. Below changes are excluded …` marker; a note belongs directly under the + version underline, before the first `~~~` header. - Confirm Phase 4d ran: no `# use next version` comment remains where the referenced provider was bumped in this wave. - **If any inter-provider `>=` floor changed** (Phase 4d resolved a pin, or a @@ -681,14 +741,18 @@ provider-by-provider: *"Provider dependency version bumps detected that should only be performed by Release Managers!"*. `git diff` the changed `pyproject.toml` files for `apache-airflow-providers-*` `>=` changes and list them for the RM. -- **Scan the new changelog sections for three entry defects** — grep the lines +- **Scan the new changelog sections for these entry defects** — grep the lines you added: (1) a bullet whose text starts with a lowercase letter → capitalize it (Phase 4b); (2) a bullet in a *visible* section (Features / Bug Fixes / Misc / Doc-only) with no `(#NNNN)` suffix → usually no-PR release-tooling that belongs in the excluded block (Phase 4b); (3) an "add then rename" pair for the same feature left as two separate entries → collapse into one net entry - naming both PRs (Phase 4b). Reviewers reliably catch all three, so fix them - before handing off. + naming both PRs (Phase 4b); (4) an internal abbreviation (`KPO`, `DFP`, `TI`, + `RTIF`, `OL`) or the phrase "system test" → rewrite in user-facing language + (Phase 4b); (5) a dependency bump that names no version, or a Dependabot group + subject of the form "… group … with N updates" → replace with the actual + packages and versions from the PR diff (Phase 4b). Reviewers reliably catch + all of these, so fix them before handing off. - Flag anything where Phase 3.5 had to escalate, so the RM can double-check. Stop here. Do not commit, do not push — the release manager opens the PR diff --git a/.github/actions/migration_tests/action.yml b/.github/actions/migration_tests/action.yml index f3e17b4b18203..9b18932028a20 100644 --- a/.github/actions/migration_tests/action.yml +++ b/.github/actions/migration_tests/action.yml @@ -39,7 +39,7 @@ runs: --use-airflow-version "${MIN_AIRFLOW_VERSION}" \ ${AIRFLOW_EXTRAS} \ --answer y && - breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANGERS} + breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANAGERS} ${AIRFLOW_3_CMD}" --no-db-cleanup env: COMPOSE_PROJECT_NAME: "docker-compose" @@ -73,7 +73,7 @@ runs: --use-airflow-version "${MIN_AIRFLOW_VERSION}" \ ${AIRFLOW_EXTRAS} \ --answer y && - breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANGERS} + breeze shell "export AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS=${DB_MANAGERS} ${AIRFLOW_3_CMD}" --no-db-cleanup env: COMPOSE_PROJECT_NAME: "docker-compose" diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 59fbedc473747..7aa3d8d6e2ec7 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -371,24 +371,6 @@ updates: - dependency-name: "*" update-types: ["version-update:semver-major"] - - package-ecosystem: "gomod" - cooldown: - default-days: 4 - directory: "/go-sdk" - schedule: - interval: "weekly" - target-branch: v3-3-test - groups: - 3-3-go-sdk-dependency-updates: - patterns: - - "*" - update-types: - - "minor" - - "patch" - ignore: - - dependency-name: "*" - update-types: ["version-update:semver-major"] - - package-ecosystem: "uv" cooldown: default-days: 4 @@ -404,17 +386,36 @@ updates: - "*" applies-to: security-updates + # Go modules resolve straight to upstream repositories: there is no central registry + # (unlike PyPI/npm) that pre-scans a release, and whoever controls a module's namespace + # can publish a version at any time. To keep an unreviewed upstream Go release from being + # auto-merged into our dev/CI chain, we do NOT raise automatic version-update PRs here + # (`open-pull-requests-limit: 0`); routine Go bumps are done manually. We still accept + # advisory-driven security updates, which require a published GHSA advisory and so cannot + # be triggered by an upstream release self-declaring itself a "security" fix. - package-ecosystem: "gomod" cooldown: - default-days: 4 + default-days: 14 directory: "/go-sdk" schedule: interval: "weekly" + open-pull-requests-limit: 0 groups: - go-sdk-dependency-updates: + go-sdk-security-updates: patterns: - "*" - go-sdk-security-updates: + applies-to: security-updates + + - package-ecosystem: "gomod" + cooldown: + default-days: 14 + directory: "/go-sdk" + schedule: + interval: "weekly" + target-branch: v3-3-test + open-pull-requests-limit: 0 + groups: + 3-3-go-sdk-security-updates: patterns: - "*" applies-to: security-updates diff --git a/.github/workflows/additional-ci-image-checks.yml b/.github/workflows/additional-ci-image-checks.yml index c294dcf0dc856..42b71e1f32bb4 100644 --- a/.github/workflows/additional-ci-image-checks.yml +++ b/.github/workflows/additional-ci-image-checks.yml @@ -28,18 +28,10 @@ on: # yamllint disable-line rule:truthy description: "Platform for the build - 'linux/amd64' or 'linux/arm64'" required: true type: string - python-versions: - description: "The list of python versions (stringified JSON array) to run the tests on." - required: true - type: string branch: description: "Branch used to run the CI jobs in (main/v*_*_test)." required: true type: string - constraints-branch: - description: "Branch used to get constraints from" - required: true - type: string default-python-version: description: "Which version of python should be used by default" required: true @@ -52,72 +44,19 @@ on: # yamllint disable-line rule:truthy description: "Whether to skip prek hooks (true/false)" required: true type: string - docker-cache: - description: "Docker cache specification to build the image (registry, local, disabled)." - required: true - type: string - disable-airflow-repo-cache: - description: "Disable airflow repo cache read from main." - required: true - type: string - canary-run: - description: "Whether this is a canary run (true/false)" - required: true - type: string latest-versions-only: description: "Whether to run only latest versions (true/false)" required: true type: string - include-success-outputs: - description: "Whether to include success outputs (true/false)" - required: true - type: string debug-resources: description: "Whether to debug resources (true/false)" required: true type: string - use-uv: - description: "Whether to use uv to build the image (true/false)" - required: true - type: string permissions: contents: read jobs: - # Push early BuildX cache to GitHub Registry in Apache repository, This cache does not wait for all the - # tests to complete - it is run very early in the build process for "main" merges in order to refresh - # cache using the current constraints. This will speed up cache refresh in cases when pyproject.toml - # changes or in case of Dockerfile changes. Failure in this step is not a problem (at most it will - # delay cache refresh. It does not attempt to upgrade to newer dependencies. - # We only push CI cache as PROD cache usually does not gain as much from fresh cache because - # it uses prepared airflow and provider distributions that invalidate the cache anyway most of the time - push-early-buildx-cache-to-github-registry: - name: Push Early Image Cache - uses: ./.github/workflows/push-image-cache.yml - permissions: - contents: read - # This write is only given here for `push` events from "apache/airflow" repo. It is not given for PRs - # from forks. This is to prevent malicious PRs from creating images in the "apache/airflow" repo. - packages: write - with: - runners: ${{ inputs.runners }} - cache-type: "Early" - include-prod-images: "false" - push-latest-images: "false" - platform: ${{ inputs.platform }} - python-versions: ${{ inputs.python-versions }} - branch: ${{ inputs.branch }} - constraints-branch: ${{ inputs.constraints-branch }} - use-uv: ${{ inputs.use-uv }} - include-success-outputs: ${{ inputs.include-success-outputs }} - docker-cache: ${{ inputs.docker-cache }} - disable-airflow-repo-cache: ${{ inputs.disable-airflow-repo-cache }} - if: > - inputs.canary-run == 'true' && - (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') - - # Check that after earlier cache push, breeze command will build quickly - # This build is a bit slow from in-the scratch builds, so we should run it only in - # regular PRs + # Check that the image builds quickly from the registry cache. This build is slow from + # scratch, so we only run it on regular PRs. check-that-image-builds-quickly: timeout-minutes: 25 name: Check that image builds quickly diff --git a/.github/workflows/airflow-e2e-tests.yml b/.github/workflows/airflow-e2e-tests.yml index 9ebb3da454ad7..a594c9990ce3c 100644 --- a/.github/workflows/airflow-e2e-tests.yml +++ b/.github/workflows/airflow-e2e-tests.yml @@ -135,7 +135,7 @@ jobs: # arch-specific auto-provisioned JDKs (the example pins a Java 11 toolchain). - name: "Setup Java for the Java SDK build" if: inputs.e2e_test_mode == 'java_sdk' - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: ${{ inputs.java-sdk-version }} @@ -175,7 +175,7 @@ jobs: e2e-go-sdk-go-v1-${{ runner.os }}-${{ runner.arch }}- - name: "Setup pnpm for the TypeScript SDK build" if: inputs.e2e_test_mode == 'ts_sdk' - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: package_json_file: ts-sdk/package.json run_install: false diff --git a/.github/workflows/asf-allowlist-check.yml b/.github/workflows/asf-allowlist-check.yml index 0c9d8697ac553..2ff06418cbfd2 100644 --- a/.github/workflows/asf-allowlist-check.yml +++ b/.github/workflows/asf-allowlist-check.yml @@ -31,4 +31,4 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: apache/infrastructure-actions/allowlist-check@0ff9972b5872e19c9f4555c9159c2fea4f794355 # main + - uses: apache/infrastructure-actions/allowlist-check@b69225a81154602226df0f441cf23b95a55cf4f8 # main diff --git a/.github/workflows/basic-tests.yml b/.github/workflows/basic-tests.yml index f7967ef1a7142..5f9810815abf0 100644 --- a/.github/workflows/basic-tests.yml +++ b/.github/workflows/basic-tests.yml @@ -115,7 +115,7 @@ jobs: - name: "Install SVN" run: sudo apt-get update && sudo apt-get install -y subversion - name: "Install Java (for Apache RAT)" - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: '17' @@ -192,7 +192,7 @@ jobs: with: persist-credentials: false - name: Setup pnpm - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: version: 9 run_install: false @@ -203,7 +203,7 @@ jobs: cache: 'pnpm' cache-dependency-path: 'airflow-core/src/airflow/**/pnpm-lock.yaml' - name: "Restore eslint cache (ui)" - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: airflow-core/src/airflow/ui/node_modules/ # yamllint disable-line rule:line-length @@ -214,7 +214,7 @@ jobs: env: FORCE_COLOR: 2 - name: "Save eslint cache (ui)" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: airflow-core/src/airflow/ui/node_modules/ key: cache-ui-node-modules-v1-${{ runner.os }}-${{ hashFiles('airflow/ui/**/pnpm-lock.yaml') }} @@ -222,7 +222,7 @@ jobs: retention-days: '2' if: steps.restore-eslint-cache-ui.outputs.stash-hit != 'true' - name: "Restore eslint cache (simple auth manager UI)" - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/node_modules/ key: > @@ -234,7 +234,7 @@ jobs: env: FORCE_COLOR: 2 - name: "Save eslint cache (ui)" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/node_modules/ key: > diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml index 4750ec2599054..7c196032e266f 100644 --- a/.github/workflows/ci-amd.yml +++ b/.github/workflows/ci-amd.yml @@ -297,19 +297,12 @@ jobs: with: runners: ${{ needs.build-info.outputs.runner-type }} platform: ${{ needs.build-info.outputs.platform }} - python-versions: ${{ needs.build-info.outputs.python-versions }} branch: ${{ needs.build-info.outputs.default-branch }} - constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} default-python-version: "${{ needs.build-info.outputs.default-python-version }}" upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }} skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }} - docker-cache: ${{ needs.build-info.outputs.docker-cache }} - disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - canary-run: ${{ needs.build-info.outputs.canary-run }} latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only }} - include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} debug-resources: ${{ needs.build-info.outputs.debug-resources }} - use-uv: ${{ needs.build-info.outputs.use-uv }} generate-constraints: name: "Generate constraints" @@ -1017,6 +1010,15 @@ jobs: - name: Run Go tests working-directory: ./go-sdk run: gotestsum --format github-actions ./... + # Go modules have no central registry that pre-scans releases, so we scan our own + # dependency graph against the Go vulnerability database. govulncheck is reachability + # based, so it only fails on advisories that actually affect code we call. The version + # tag is immutable (Go modules are content-addressed via the checksum database). + - name: Scan Go dependencies for known vulnerabilities + working-directory: ./go-sdk + run: | + go install golang.org/x/vuln/cmd/govulncheck@v1.6.0 + govulncheck ./... tests-java-sdk: name: "Java SDK tests" @@ -1038,7 +1040,7 @@ jobs: with: persist-credentials: false - name: Setup Java - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: ${{ env.JAVA_VERSION }} diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml index 35344fdcccd10..3515240ee4df3 100644 --- a/.github/workflows/ci-arm.yml +++ b/.github/workflows/ci-arm.yml @@ -286,19 +286,12 @@ jobs: with: runners: ${{ needs.build-info.outputs.runner-type }} platform: ${{ needs.build-info.outputs.platform }} - python-versions: ${{ needs.build-info.outputs.python-versions }} branch: ${{ needs.build-info.outputs.default-branch }} - constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} default-python-version: "${{ needs.build-info.outputs.default-python-version }}" upgrade-to-newer-dependencies: ${{ needs.build-info.outputs.upgrade-to-newer-dependencies }} skip-prek-hooks: ${{ needs.build-info.outputs.skip-prek-hooks }} - docker-cache: ${{ needs.build-info.outputs.docker-cache }} - disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} - canary-run: ${{ needs.build-info.outputs.canary-run }} latest-versions-only: ${{ needs.build-info.outputs.latest-versions-only }} - include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} debug-resources: ${{ needs.build-info.outputs.debug-resources }} - use-uv: ${{ needs.build-info.outputs.use-uv }} generate-constraints: name: "Generate constraints" @@ -1006,6 +999,15 @@ jobs: - name: Run Go tests working-directory: ./go-sdk run: gotestsum --format github-actions ./... + # Go modules have no central registry that pre-scans releases, so we scan our own + # dependency graph against the Go vulnerability database. govulncheck is reachability + # based, so it only fails on advisories that actually affect code we call. The version + # tag is immutable (Go modules are content-addressed via the checksum database). + - name: Scan Go dependencies for known vulnerabilities + working-directory: ./go-sdk + run: | + go install golang.org/x/vuln/cmd/govulncheck@v1.6.0 + govulncheck ./... tests-java-sdk: name: "Java SDK tests" @@ -1027,7 +1029,7 @@ jobs: with: persist-credentials: false - name: Setup Java - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: ${{ env.JAVA_VERSION }} diff --git a/.github/workflows/ci-image-build.yml b/.github/workflows/ci-image-build.yml index a0e8f1d40fa17..fcd53306bc863 100644 --- a/.github/workflows/ci-image-build.yml +++ b/.github/workflows/ci-image-build.yml @@ -167,7 +167,7 @@ jobs: - name: > Restore the commit the CI image stashed for ref ${{ inputs.image-stash-ref }} was built from - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-image-commit-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ -${{ inputs.image-stash-ref }}" @@ -200,7 +200,7 @@ jobs: - name: > Restore CI docker image built for ref ${{ inputs.image-stash-ref }} ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }} - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ -${{ inputs.image-stash-ref }}" @@ -215,7 +215,7 @@ jobs: - name: > Restore ci-cache mount image ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }} ${{ inputs.image-stash-ref != '' && format('for ref {0}', inputs.image-stash-ref) || '' }} - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-cache-mount-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ ${{ inputs.image-stash-ref != '' && format('-{0}', inputs.image-stash-ref) || '' }}" @@ -242,7 +242,7 @@ jobs: --cache-file /tmp/ci-cache-mount-save-v3-${PYTHON_MAJOR_MINOR_VERSION}.tar.gz if: steps.restore-cache-mount.outputs.stash-hit == 'true' - name: "Restore CI docker image ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }}" - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}" path: "/mnt/" @@ -327,7 +327,7 @@ jobs: shell: bash if: inputs.upload-image-artifact == 'true' && steps.stashed-image.outputs.reusable != 'true' - name: "Stash CI docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: ci-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }} path: "/mnt/ci-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar" @@ -340,7 +340,7 @@ jobs: # what a later publish of that same ref reads - hence a retention that outlives the shared # one, the same ref being rebuilt three days apart at the very least. - name: "Stash CI docker image built for ref ${{ inputs.image-stash-ref }}" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ -${{ inputs.image-stash-ref }}" @@ -353,7 +353,7 @@ jobs: # Saved last and with the image's retention, so that finding this commit is enough to # know the image it describes is there to be restored. - name: "Stash the commit the CI image for ref ${{ inputs.image-stash-ref }} was built from" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-image-commit-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ -${{ inputs.image-stash-ref }}" @@ -375,7 +375,7 @@ jobs: - name: > Stash cache mount ${{ inputs.platform }}:${{ env.PYTHON_MAJOR_MINOR_VERSION }} ${{ inputs.image-stash-ref != '' && format('for ref {0}', inputs.image-stash-ref) || '' }} - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: "ci-cache-mount-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }}\ ${{ inputs.image-stash-ref != '' && format('-{0}', inputs.image-stash-ref) || '' }}" diff --git a/.github/workflows/ci-image-checks.yml b/.github/workflows/ci-image-checks.yml index 160032767a843..6d81d85dff9d9 100644 --- a/.github/workflows/ci-image-checks.yml +++ b/.github/workflows/ci-image-checks.yml @@ -206,7 +206,7 @@ jobs: use-uv: ${{ inputs.use-uv }} make-mnt-writeable-and-cleanup: true - name: "Restore docs inventory cache" - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: ./generated/_inventory_cache/ key: cache-docs-inventory-v1 @@ -322,7 +322,7 @@ jobs: env: SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} - name: "Save docs inventory cache" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: ./generated/_inventory_cache/ key: cache-docs-inventory-v1 diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 6c5a975199e6c..a073ab4488705 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -130,19 +130,19 @@ jobs: - name: Setup Java if: matrix.language == 'java' - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: '11' - name: Initialize CodeQL - uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/init@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5 with: languages: ${{ matrix.language }} - name: Autobuild if: matrix.language != 'java' - uses: github/codeql-action/autobuild@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/autobuild@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5 - name: Build Java SDK if: matrix.language == 'java' @@ -150,7 +150,7 @@ jobs: run: ./gradlew classes testClasses - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3 + uses: github/codeql-action/analyze@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5 with: # Provide more context to the SARIF output (shows up in run.automationDetails.id field) category: "/language:${{matrix.language}}" diff --git a/.github/workflows/generate-constraints.yml b/.github/workflows/generate-constraints.yml index 28c078a356347..026ad45401a9a 100644 --- a/.github/workflows/generate-constraints.yml +++ b/.github/workflows/generate-constraints.yml @@ -44,6 +44,24 @@ on: # yamllint disable-line rule:truthy description: "Whether to generate PyPI constraints (true/false)" required: true type: string + allow-pre-releases: + description: > + Whether the PyPI constraints may pin pre-release providers (true/false). Set when + constraints are cut for a release candidate, whose providers are on PyPI only as rc + versions. Scoped to apache-airflow-providers-* - no other package can resolve to a + pre-release. + required: false + default: "false" + type: string + pypi-providers-only: + description: > + Resolve against providers exactly as published on PyPI (true/false). Skips building the + provider distributions from sources and the source-providers constraints with them, so + nothing local can answer for a provider. Set when the constraints being produced are the + ones a release ships - see the comments on the steps this disables. + required: false + default: "false" + type: string debug-resources: description: "Whether to run in debug mode (true/false)" required: true @@ -104,12 +122,15 @@ jobs: python: ${{ matrix.python-version }} use-uv: ${{ inputs.use-uv }} make-mnt-writeable-and-cleanup: true + # Resolves the providers from the sources in this checkout. Skipped for a release, which + # ships only the PyPI constraints and must not have a source tree answer for a provider. - name: "Source constraints" shell: bash run: > breeze release-management generate-constraints --airflow-constraints-mode constraints-source-providers --answer yes --python "${PYTHON_VERSION}" + if: inputs.pypi-providers-only != 'true' - name: "No providers constraints" shell: bash timeout-minutes: 25 @@ -118,12 +139,16 @@ jobs: --airflow-constraints-mode constraints-no-providers --answer yes --python "${PYTHON_VERSION}" if: inputs.generate-no-providers-constraints == 'true' + # Builds the providers from sources into the dist directory the PyPI resolution reads through + # `--find-links`, so a locally built wheel can answer for a provider and is then excluded from + # the constraints. That is wanted while developing a provider that is not on PyPI yet; it is the + # opposite of what a release wants, which is every provider pinned at its published version. - name: "Prepare updated provider distributions" shell: bash run: > breeze release-management prepare-provider-distributions --include-not-ready-providers --distribution-format wheel - if: inputs.generate-pypi-constraints == 'true' + if: inputs.generate-pypi-constraints == 'true' && inputs.pypi-providers-only != 'true' - name: "Prepare airflow distributions" shell: bash run: > @@ -137,9 +162,14 @@ jobs: - name: "PyPI constraints" shell: bash timeout-minutes: 25 + env: + # Deliberately not named ALLOW_PRE_RELEASES: that is the option's own envvar, and having + # both paths set it would make it unclear which one is in force. Empty on a normal run, + # so the unquoted expansion below contributes no argument at all. + PRE_RELEASE_FLAG: ${{ inputs.allow-pre-releases == 'true' && '--allow-pre-releases' || '' }} run: | breeze release-management generate-constraints --airflow-constraints-mode constraints \ - --answer yes --python "${PYTHON_VERSION}" + --answer yes --python "${PYTHON_VERSION}" ${PRE_RELEASE_FLAG} if: inputs.generate-pypi-constraints == 'true' - name: "Check for provider downgrade alert" id: downgrade-alert diff --git a/.github/workflows/java-sdk-release-verify.yml b/.github/workflows/java-sdk-release-verify.yml index 627d6d4a71e90..9e170ed1787df 100644 --- a/.github/workflows/java-sdk-release-verify.yml +++ b/.github/workflows/java-sdk-release-verify.yml @@ -76,7 +76,7 @@ jobs: fetch-depth: 0 persist-credentials: false - name: Set up JDK ${{ env.JAVA_VERSION }} - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: temurin java-version: ${{ env.JAVA_VERSION }} @@ -125,7 +125,7 @@ jobs: fetch-depth: 0 persist-credentials: false - name: Set up JDK ${{ env.JAVA_VERSION }} - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: temurin java-version: ${{ env.JAVA_VERSION }} diff --git a/.github/workflows/k8s-tests.yml b/.github/workflows/k8s-tests.yml index 0a14493f07842..55569c452af1d 100644 --- a/.github/workflows/k8s-tests.yml +++ b/.github/workflows/k8s-tests.yml @@ -203,7 +203,7 @@ jobs: restore-keys: | lang-sdk-go-v1-${{ runner.os }}-${{ runner.arch }}- - name: "Setup Java for lang-SDK build" - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0 + uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0 with: distribution: 'temurin' java-version: ${{ inputs.java-sdk-version }} diff --git a/.github/workflows/prod-image-build.yml b/.github/workflows/prod-image-build.yml index 838b7f2eba975..a51add8321837 100644 --- a/.github/workflows/prod-image-build.yml +++ b/.github/workflows/prod-image-build.yml @@ -284,7 +284,7 @@ jobs: breeze prod-image save --platform "${PLATFORM}" --image-file-dir "/mnt" if: inputs.upload-image-artifact == 'true' - name: "Stash PROD docker image ${{ env.PYTHON_MAJOR_MINOR_VERSION }}" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 with: key: prod-image-save-v3-${{ inputs.platform }}-${{ env.PYTHON_MAJOR_MINOR_VERSION }} path: "/mnt/prod-image-save-*-${{ env.PYTHON_MAJOR_MINOR_VERSION }}.tar" diff --git a/.github/workflows/publish-docs-to-s3.yml b/.github/workflows/publish-docs-to-s3.yml index 1c613209794b4..759956e03b0a1 100644 --- a/.github/workflows/publish-docs-to-s3.yml +++ b/.github/workflows/publish-docs-to-s3.yml @@ -344,7 +344,7 @@ jobs: ${{ secrets.CONSTRAINTS_GITHUB_REPOSITORY != '' && secrets.CONSTRAINTS_GITHUB_REPOSITORY || 'apache/airflow' }} - name: "Restore docs inventory cache" - uses: apache/infrastructure-actions/stash/restore@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/restore@b69225a81154602226df0f441cf23b95a55cf4f8 with: path: ./generated/_inventory_cache/ key: cache-docs-inventory-v1 @@ -374,7 +374,7 @@ jobs: breeze build-docs ${SPHINX_INCLUDE_DOCS} --docs-only ${FAIL_ON_INVENTORIES} fi - name: "Save docs inventory cache" - uses: apache/infrastructure-actions/stash/save@0ff9972b5872e19c9f4555c9159c2fea4f794355 + uses: apache/infrastructure-actions/stash/save@b69225a81154602226df0f441cf23b95a55cf4f8 if: >- steps.restore-docs-inventory-cache.outputs.stash-hit != 'true' && steps.build-sphinx-docs.outputs.sphinx-build-skipped != 'true' diff --git a/.github/workflows/refresh-constraints.yml b/.github/workflows/refresh-constraints.yml new file mode 100644 index 0000000000000..58295b6c092e0 --- /dev/null +++ b/.github/workflows/refresh-constraints.yml @@ -0,0 +1,72 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Refresh constraints +# Refreshes the pinned constraint files on demand, when waiting for the automatic path is not an +# option -- a `main` / `vX-Y-test` build that cannot be made green, or newly released providers +# that a release candidate has to pick up before it is promoted. +# +# All three constraint flavours are refreshed, not just the PyPI one: +# +# * `constraints-source-providers-X.Y.txt` -- providers resolved from this checkout's sources, +# which is what a contributor's Breeze environment and CI install from; +# * `constraints-no-providers-X.Y.txt` -- core dependencies alone, used to install Airflow +# without any provider pinned; +# * `constraints-X.Y.txt` -- providers as published on PyPI, which is what users install with. +# +# Refreshing only the PyPI flavour would leave CI and Breeze on the older pins, so the three +# would disagree about the same dependency. +# +# This does the same work as the automatic `uv.lock` push (whose workflow it calls), against the +# ref you name rather than the branch the push landed on. Run it from `main`: the workflow +# definition and `breeze` come from `main`, while the sources come from `ref`, so refreshing a +# release line needs no cherry-pick to `vX-Y-test` / `vX-Y-stable` first. The `constraints-X-Y` +# branch written to is derived from that ref's `branch_defaults.py`. +# +# See dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md. +on: # yamllint disable-line rule:truthy + workflow_dispatch: + inputs: + ref: + description: "Repo reference to refresh constraints from (e.g. v3-3-stable). Empty = this branch." + required: false + default: "" + type: string + upgrade-to-newer-dependencies: + description: "Upgrade deps to newest from PyPI" + required: false + type: boolean + default: true +permissions: + contents: read +concurrency: + # Two refreshes of the same ref would race each other onto the same `constraints-X-Y` branch. + group: refresh-constraints-${{ inputs.ref || github.ref }} + cancel-in-progress: false +jobs: + refresh-constraints: + name: "Refresh constraints" + uses: ./.github/workflows/update-constraints-on-push.yml + permissions: + contents: write + packages: write + with: + ref: ${{ inputs.ref }} + upgrade-to-newer-dependencies: ${{ inputs.upgrade-to-newer-dependencies }} + secrets: + SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} diff --git a/.github/workflows/refresh-image-registry-cache.yml b/.github/workflows/refresh-image-registry-cache.yml new file mode 100644 index 0000000000000..23b3e6221dc6d --- /dev/null +++ b/.github/workflows/refresh-image-registry-cache.yml @@ -0,0 +1,145 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Refresh image registry cache +# Refreshes the CI image cache in the GitHub registry, in a run of its own. +# +# It has to be its own workflow rather than a job inside `ci-amd.yml` / `ci-arm.yml`, because a +# job cannot opt out of its run being cancelled, and those runs are cancelled by the next push. +# With no cache a branch builds cold (~20 minutes per Python version), which widens the window in +# which the next push cancels the run, which leaves the cache missing. `v3-3-test` sat in that +# loop -- its `linux/amd64` cache covered the default Python version alone, the single matrix +# entry that finished before the next push killed the other four seconds later. Being a separate +# workflow is what lets a manual run escape that (see the concurrency group below). +# +# Every Python version is refreshed in both trigger paths. That is what makes the cache useful: +# a version with no cache entry builds from scratch on every run, whichever way it got skipped. +on: # yamllint disable-line rule:truthy + workflow_dispatch: + inputs: + platform: + description: "Platform to refresh the cache for" + type: choice + options: + - "both" + - "linux/amd64" + - "linux/arm64" + default: "both" + push: + # Release-prep and providers branches only -- they are built on push, so the CI run that + # would have refreshed their cache is also the run that keeps being cancelled. `main` is not + # built on push; its cache is refreshed by the Push Image Cache job at the end of the + # scheduled canary, which runs to completion. + branches: + - v[0-9]+-[0-9]+-test + - providers-[a-z]+-?[a-z]*/v[0-9]+-[0-9]+ +permissions: + contents: read +concurrency: + # Push runs are never cancelled -- being cancelled by the next merge is the whole failure this + # workflow exists to escape, so a newer one queues behind the one in flight rather than + # replacing it. GitHub keeps at most one run pending per group, so the queue cannot grow. + # + # Manual runs are grouped separately, which is what keeps a merge from cancelling one, and they + # do supersede each other: firing the workflow again for the same branch says the run in flight + # is not the one you want. + group: refresh-image-registry-cache-${{ github.ref }}-${{ github.event_name }} + cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }} +jobs: + build-info: + name: "Build info" + # No outputs yet at this stage, so the runner has to be hard-coded to a public one. + runs-on: ["ubuntu-22.04"] + outputs: + default-branch: ${{ steps.selective-checks.outputs.default-branch }} + default-constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }} + disable-airflow-repo-cache: ${{ steps.selective-checks.outputs.disable-airflow-repo-cache }} + docker-cache: ${{ steps.selective-checks.outputs.docker-cache }} + include-success-outputs: ${{ steps.selective-checks.outputs.include-success-outputs }} + python-versions: ${{ steps.selective-checks.outputs.python-versions }} + use-uv: ${{ steps.selective-checks.outputs.force-pip == 'true' && 'false' || 'true' }} + steps: + - name: "Cleanup repo" + shell: bash + run: sudo rm -rf ${GITHUB_WORKSPACE}/* + - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: "Install Breeze" + uses: ./.github/actions/breeze + - name: Selective checks + id: selective-checks + env: + # Forces every Python version. Selective checks narrows `python-versions` down to the + # default one on a push where only text files changed + # (`_should_run_all_tests_and_versions`), which would leave the rest of the versions + # with no cache -- the failure mode this workflow exists to prevent. + PR_LABELS: '["all versions"]' + COMMIT_REF: "${{ github.sha }}" + VERBOSE: "false" + run: breeze ci selective-check 2>> ${GITHUB_OUTPUT} + + refresh-ci-cache-amd: + name: "Refresh cache linux/amd64" + needs: [build-info] + uses: ./.github/workflows/push-image-cache.yml + permissions: + contents: read + # Only granted for `push` / `workflow_dispatch` on apache/airflow itself, never for forks. + packages: write + with: + runners: '["ubuntu-22.04"]' + platform: "linux/amd64" + cache-type: "Refresh" + # The PROD cache is built from distributions prepared earlier in a CI run, and a standalone + # refresh has no such artifact. The CI cache is also where the build time goes. + include-prod-images: "false" + # `latest` is what a build pulls when it has no cache at all. Moving it belongs to a canary + # run that has passed its tests, not to a cache refresh. + push-latest-images: "false" + python-versions: ${{ needs.build-info.outputs.python-versions }} + branch: ${{ needs.build-info.outputs.default-branch }} + constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} + use-uv: ${{ needs.build-info.outputs.use-uv }} + include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} + docker-cache: ${{ needs.build-info.outputs.docker-cache }} + disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} + if: github.event_name == 'push' || inputs.platform == 'both' || inputs.platform == 'linux/amd64' + + refresh-ci-cache-arm: + name: "Refresh cache linux/arm64" + needs: [build-info] + uses: ./.github/workflows/push-image-cache.yml + permissions: + contents: read + packages: write + with: + runners: '["ubuntu-22.04-arm"]' + platform: "linux/arm64" + cache-type: "Refresh" + include-prod-images: "false" + push-latest-images: "false" + python-versions: ${{ needs.build-info.outputs.python-versions }} + branch: ${{ needs.build-info.outputs.default-branch }} + constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} + use-uv: ${{ needs.build-info.outputs.use-uv }} + include-success-outputs: ${{ needs.build-info.outputs.include-success-outputs }} + docker-cache: ${{ needs.build-info.outputs.docker-cache }} + disable-airflow-repo-cache: ${{ needs.build-info.outputs.disable-airflow-repo-cache }} + if: github.event_name == 'push' || inputs.platform == 'both' || inputs.platform == 'linux/arm64' diff --git a/.github/workflows/registry-backfill.yml b/.github/workflows/registry-backfill.yml index 298a9e1ce043c..368c99301cfef 100644 --- a/.github/workflows/registry-backfill.yml +++ b/.github/workflows/registry-backfill.yml @@ -218,7 +218,7 @@ jobs: --provider "${PROVIDER}" ${VERSION_ARGS} - name: "Setup pnpm" - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: version: 10 diff --git a/.github/workflows/registry-build.yml b/.github/workflows/registry-build.yml index da239882629a1..1de3fd22e142e 100644 --- a/.github/workflows/registry-build.yml +++ b/.github/workflows/registry-build.yml @@ -277,7 +277,7 @@ jobs: fi - name: "Setup pnpm" - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: version: 10 diff --git a/.github/workflows/release-constraints.yml b/.github/workflows/release-constraints.yml new file mode 100644 index 0000000000000..d815251380f16 --- /dev/null +++ b/.github/workflows/release-constraints.yml @@ -0,0 +1,277 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Resolves the constraints that a release ships, rather than tagging whatever the +# `constraints-X-Y` branch happened to hold when the release ran. +# +# The stage is derived from the version, so the two cannot be mismatched by hand: +# +# * a candidate (`3.3.1rc1`) pins the providers at the versions PyPI holds - the wave being +# voted on exists there only as rc versions - and lands on a branch of its own, so a candidate +# never moves the branch every other build reads. Re-running it for the same candidate +# replaces that branch and its tag, so the two always describe the run that produced them; +# * a final (`3.3.1`) pins the providers at their released versions, ignoring any candidate, and +# commits onto `constraints-X-Y` itself, which is what makes the released constraints the +# baseline everything downstream reads. +# +# A final therefore cannot be produced by retagging a candidate: a released version must never +# pin an rc. +--- +name: Release constraints +on: # yamllint disable-line rule:truthy + workflow_dispatch: + inputs: + version: + description: "Version the constraints belong to, e.g. 3.3.1rc1 or 3.3.1" + required: true + type: string + ref: + description: "Ref the constraints are resolved from, e.g. v3-3-stable or the release tag" + required: true + type: string +permissions: + contents: read +concurrency: + group: release-constraints-${{ inputs.version }} + cancel-in-progress: false +jobs: + build-info: + timeout-minutes: 10 + name: "Build info" + runs-on: ["ubuntu-22.04"] + if: contains(fromJSON('[ + "ashb", + "eladkal", + "ephraimbuddy", + "jedcunningham", + "kaxil", + "pierrejeambrun", + "potiuk", + "utkarsharma2", + "vatsrahul1001", + "vincbeck", + ]'), github.event.sender.login) + outputs: + python-versions: ${{ steps.selective-checks.outputs.python-versions }} + python-versions-list-as-string: ${{ steps.selective-checks.outputs.python-versions-list-as-string }} + default-branch: ${{ steps.selective-checks.outputs.default-branch }} + default-constraints-branch: ${{ steps.selective-checks.outputs.default-constraints-branch }} + constraints-branch: ${{ steps.stage.outputs.constraints-branch }} + target-branch: ${{ steps.stage.outputs.target-branch }} + allow-pre-releases: ${{ steps.stage.outputs.allow-pre-releases }} + steps: + - name: "Cleanup repo" + shell: bash + run: sudo rm -rf ${GITHUB_WORKSPACE}/* + - name: "Checkout ${{ inputs.ref }}" + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.ref }} + fetch-depth: 2 + persist-credentials: false + - name: "Install Breeze" + uses: ./.github/actions/breeze + id: breeze + - name: "Save github context to file" + # See ci-amd.yml for the full rationale: avoids ARG_MAX on big PRs by writing the + # github context to a file, and the single-quoted heredoc makes the zizmor + # template-injection finding a false positive (no bash expansion happens inside it). + shell: bash + run: | # zizmor: ignore[template-injection] + cat > "${RUNNER_TEMP}/github_context.json" << '__GITHUB_CONTEXT_END__' + ${{ toJson(github) }} + __GITHUB_CONTEXT_END__ + - name: Selective checks + id: selective-checks + env: + PR_LABELS: "[]" + COMMIT_REF: "${{ inputs.ref }}" + VERBOSE: "false" + GITHUB_CONTEXT_INPUT: "${{ runner.temp }}/github_context.json" + run: breeze ci selective-check 2>> ${GITHUB_OUTPUT} + - name: "Derive the release stage from the version" + id: stage + shell: bash + env: + VERSION: ${{ inputs.version }} + run: | + if [[ ! "${VERSION}" =~ ^[0-9]+\.[0-9]+\.[0-9]+(rc[0-9]+)?$ ]]; then + echo "'${VERSION}' is not a release version - expected X.Y.Z or X.Y.ZrcN." >&2 + exit 1 + fi + major="$(echo "${VERSION}" | cut -d. -f1)" + minor="$(echo "${VERSION}" | cut -d. -f2)" + constraints_branch="constraints-${major}-${minor}" + echo "constraints-branch=${constraints_branch}" >> "${GITHUB_OUTPUT}" + if [[ "${VERSION}" == *rc* ]]; then + echo "allow-pre-releases=true" >> "${GITHUB_OUTPUT}" + echo "target-branch=constraints-${VERSION}" >> "${GITHUB_OUTPUT}" + else + echo "allow-pre-releases=false" >> "${GITHUB_OUTPUT}" + echo "target-branch=${constraints_branch}" >> "${GITHUB_OUTPUT}" + fi + - name: "Parameters summary" + shell: bash + env: + VERSION: ${{ inputs.version }} + REF: ${{ inputs.ref }} + CONSTRAINTS_BRANCH: ${{ steps.stage.outputs.constraints-branch }} + TARGET_BRANCH: ${{ steps.stage.outputs.target-branch }} + ALLOW_PRE_RELEASES: ${{ steps.stage.outputs.allow-pre-releases }} + run: | + { + echo "## Release constraints" + echo "" + echo "| Parameter | Value |" + echo "|---|---|" + echo "| Version | \`${VERSION}\` |" + echo "| Resolved from ref | \`${REF}\` |" + echo "| Branched off | \`${CONSTRAINTS_BRANCH}\` |" + echo "| Committed to | \`${TARGET_BRANCH}\` |" + echo "| Pre-releases allowed | \`${ALLOW_PRE_RELEASES}\` |" + echo "| Tagged | \`constraints-${VERSION}\` |" + } | tee -a "${GITHUB_STEP_SUMMARY}" + + build-ci-images: + name: "Build CI images" + needs: [build-info] + uses: ./.github/workflows/ci-image-build.yml + permissions: + contents: read + packages: write + with: + runners: '["ubuntu-22.04"]' + platform: "linux/amd64" + push-image: "false" + upload-image-artifact: "true" + upload-mount-cache-artifact: "false" + python-versions: ${{ needs.build-info.outputs.python-versions }} + branch: ${{ needs.build-info.outputs.default-branch }} + constraints-branch: ${{ needs.build-info.outputs.default-constraints-branch }} + checkout-ref: ${{ inputs.ref }} + use-uv: "true" + upgrade-to-newer-dependencies: "false" + docker-cache: "registry" + disable-airflow-repo-cache: "false" + + generate-constraints: + name: "Generate constraints" + needs: [build-info, build-ci-images] + uses: ./.github/workflows/generate-constraints.yml + with: + runners: '["ubuntu-22.04"]' + platform: "linux/amd64" + python-versions-list-as-string: ${{ needs.build-info.outputs.python-versions-list-as-string }} + python-versions: ${{ needs.build-info.outputs.python-versions }} + generate-pypi-constraints: "true" + # Only the PyPI constraints are what a release ships; the other modes serve CI, and + # regenerating them here would move them for reasons unrelated to the release. + generate-no-providers-constraints: "false" + # A release pins the providers as published on PyPI, so nothing is built from the sources + # here: a locally built provider wheel would answer the resolution and then be left out of + # the constraints, which is exactly the version the release needed pinned. + pypi-providers-only: "true" + allow-pre-releases: ${{ needs.build-info.outputs.allow-pre-releases }} + debug-resources: "false" + checkout-ref: ${{ inputs.ref }} + use-uv: "true" + secrets: + SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }} + + publish-constraints: + runs-on: ["ubuntu-22.04"] + timeout-minutes: 20 + name: "Publish and tag constraints" + needs: [build-info, generate-constraints] + permissions: + contents: write + packages: read + env: + VERSION: ${{ inputs.version }} + CONSTRAINTS_BRANCH: ${{ needs.build-info.outputs.constraints-branch }} + TARGET_BRANCH: ${{ needs.build-info.outputs.target-branch }} + ALLOW_PRE_RELEASES: ${{ needs.build-info.outputs.allow-pre-releases }} + steps: + - name: "Cleanup repo" + shell: bash + run: sudo rm -rf ${GITHUB_WORKSPACE}/* + - name: "Checkout ${{ inputs.ref }}" + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.ref }} + persist-credentials: false + # Branched off the shared constraints branch in both cases; what differs is where the commit + # ends up, which the checkout below decides. + - name: "Checkout ${{ needs.build-info.outputs.constraints-branch }}" + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: "constraints" + ref: ${{ needs.build-info.outputs.constraints-branch }} + persist-credentials: true + fetch-depth: 0 + - name: "Download constraints from the generate-constraints job" + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: constraints-* + path: ./files + # A candidate's branch and tag belong to that candidate alone, so re-running for the same + # rc replaces them rather than adding to them: the branch would otherwise already hold the + # previous run's constraints (making the push a non-fast-forward) and the tag already exist. + # A final never gets this - it commits onto the shared constraints-X-Y branch, whose history + # every other build reads. + - name: "Delete the previous ${{ needs.build-info.outputs.target-branch }} branch and tag" + if: needs.build-info.outputs.allow-pre-releases == 'true' + working-directory: "constraints" + shell: bash + run: | + if git ls-remote --exit-code origin "refs/heads/${TARGET_BRANCH}" > /dev/null; then + echo "Deleting the existing '${TARGET_BRANCH}' branch." + git push origin --delete "refs/heads/${TARGET_BRANCH}" + fi + if git ls-remote --exit-code origin "refs/tags/constraints-${VERSION}" > /dev/null; then + echo "Deleting the existing 'constraints-${VERSION}' tag." + git push origin --delete "refs/tags/constraints-${VERSION}" + fi + git tag --delete "constraints-${VERSION}" > /dev/null 2>&1 || true + git branch --delete --force "${TARGET_BRANCH}" > /dev/null 2>&1 || true + - name: "Switch to ${{ needs.build-info.outputs.target-branch }}" + working-directory: "constraints" + shell: bash + run: git switch -c "${TARGET_BRANCH}" 2>/dev/null || git switch "${TARGET_BRANCH}" + - name: "Diff in constraints for Python: ${{ needs.build-info.outputs.python-versions-list-as-string }}" + run: ./scripts/ci/constraints/ci_diff_constraints.sh + - name: "Commit changed constraint files" + run: ./scripts/ci/constraints/ci_commit_constraints.sh + - name: "Push ${{ needs.build-info.outputs.target-branch }}" + working-directory: "constraints" + shell: bash + # Qualify as refs/heads/ -- for a release candidate the branch and the tag share a name + # (both constraints-), so a bare refspec is ambiguous ("matches more than one"). + run: git push origin "refs/heads/${TARGET_BRANCH}" + - name: "Tag constraints-${{ inputs.version }}" + working-directory: "constraints" + shell: bash + run: | + git tag -a "constraints-${VERSION}" -m "Constraints for Apache Airflow ${VERSION}" + git push origin "refs/tags/constraints-${VERSION}" + - name: "Summary" + shell: bash + run: | + { + echo "Constraints for \`${VERSION}\` are on \`${TARGET_BRANCH}\`," + echo "tagged \`constraints-${VERSION}\`." + } | tee -a "${GITHUB_STEP_SUMMARY}" diff --git a/.github/workflows/ui-e2e-tests.yml b/.github/workflows/ui-e2e-tests.yml index 4699aab04f188..e1654e1f152ac 100644 --- a/.github/workflows/ui-e2e-tests.yml +++ b/.github/workflows/ui-e2e-tests.yml @@ -121,7 +121,7 @@ jobs: uses: ./.github/actions/breeze if: github.event_name == 'workflow_dispatch' - name: "Setup pnpm" - uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9 + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: version: 9 run_install: false diff --git a/.github/workflows/update-constraints-on-push.yml b/.github/workflows/update-constraints-on-push.yml index e7d273f319b20..c0fc95b3e16b6 100644 --- a/.github/workflows/update-constraints-on-push.yml +++ b/.github/workflows/update-constraints-on-push.yml @@ -16,14 +16,15 @@ # under the License. # --- -name: Update constraints (on uv.lock push or manual dispatch) +name: Update constraints (on uv.lock push) # This workflow refreshes the pinned constraint files stored in the # `constraints-*` branches. It runs automatically whenever `uv.lock` changes on -# `main` or a `vX-Y-test` branch, and can also be triggered manually via the -# "Run workflow" button (workflow_dispatch) - for example to pick up newly -# released providers/dependencies from PyPI just before promoting an RC. +# `main` or a `vX-Y-test` branch. The manual path lives in +# `refresh-constraints.yml`, which calls this workflow -- keeping the two triggers +# in separate workflows means a manual refresh is not filtered by `paths: uv.lock` +# and shows up in the Actions list under its own name. # -# The manual run is always launched from `main` (so the workflow definition and +# A manual run is always launched from `main` (so the workflow definition and # `breeze` come from `main`), and the `ref` input selects the commit-ish # (branch, tag or commit hash) whose sources the constraints are refreshed from. # The `constraints-X-Y` branch to push to is derived from that ref's @@ -37,7 +38,7 @@ on: # yamllint disable-line rule:truthy - v[0-9]+-[0-9]+-test paths: - 'uv.lock' - workflow_dispatch: + workflow_call: inputs: # `ref` is optional. Leave it empty to refresh constraints for the branch # you run the workflow from (normally `main` -> `constraints-main`). Set a @@ -51,8 +52,13 @@ on: # yamllint disable-line rule:truthy type: string upgrade-to-newer-dependencies: description: "Upgrade deps to newest from PyPI" + required: false type: boolean default: true + secrets: + SLACK_BOT_TOKEN: + description: "Slack bot token used to post provider-downgrade alerts (optional)." + required: false permissions: contents: read diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 038d8ebc80e38..399bd0d335a75 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -605,7 +605,7 @@ repos: types_or: [python, pyi] args: [--fix] require_serial: true - additional_dependencies: ['ruff==0.15.22'] + additional_dependencies: ['ruff==0.16.0'] exclude: ^airflow-core/tests/unit/dags/test_imports\.py$|^performance/tests/test_.*\.py$ - id: ruff-format name: Run 'ruff format' diff --git a/AGENTS.md b/AGENTS.md index 0c8829d79b65a..2086b4040ad15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,7 +52,7 @@ Don't spell out **Directed Acyclic Graph** except for historical context. - **Lint with ruff only:** `prek run ruff --from-ref ` - **Format with ruff only:** `prek run ruff-format --from-ref ` - **Run regular (fast) static checks:** `prek run --from-ref --stage pre-commit` -- **Run manual (slower) checks:** `prek run --from-ref --stage manual` +- **Run manual (slower) checks:** `prek run --from-ref --stage manual --skip compile-ui-assets-dev --skip view-skill-eval` (the skipped hooks start long-running local servers rather than checks that complete) - **Build docs:** `breeze build-docs` - **Determine which tests to run based on changed files:** `breeze ci selective-check --commit-ref ` @@ -294,7 +294,10 @@ code review checklist in [`.github/instructions/code-review.instructions.md`](.g described in this file. 4. Run regular (fast) static checks (`prek run --from-ref --stage pre-commit`) and fix any failures. This includes mypy checks for non-provider projects (airflow-core, task-sdk, airflow-ctl, dev, scripts, devel-common). -5. Run manual (slower) checks (`prek run --from-ref --stage manual`) and fix any failures. +5. Run manual (slower) checks + (`prek run --from-ref --stage manual --skip compile-ui-assets-dev --skip view-skill-eval`) + and fix any failures. The skipped hooks start long-running local servers rather than checks + that complete. 6. Run relevant individual tests and confirm they pass. 7. Find which tests to run for the changes with selective-checks and run those tests in parallel to confirm they pass and check for CI-specific issues. 8. Check for security issues — no secrets, no injection vulnerabilities, no unsafe patterns. diff --git a/README.md b/README.md index 520deae0c95ab..bdc600e5e5011 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ Use Airflow to author workflows (Dags) that orchestrate tasks. The Airflow sched Airflow works best with workflows that are mostly static and slowly changing. When the Dag structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](https://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). +Beyond traditional data pipelines, Airflow is widely used to orchestrate machine learning workflows — training, retraining, evaluation, and deployment — and increasingly to orchestrate agentic and LLM-based workloads, coordinating the steps of an AI pipeline (data prep, tool calls, model invocation, evaluation) rather than acting as the agent itself. This isn't a new direction: teams have run ML and AI workloads on Airflow for years, and the ecosystem of providers supporting these use cases (see the [AI & ML section of the Airflow registry](https://airflow.apache.org/registry/providers/?category=ai-ml)) continues to grow. + Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [XCom feature](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html)). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work. Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches. diff --git a/airflow-core/docs/core-concepts/dag-run.rst b/airflow-core/docs/core-concepts/dag-run.rst index b536681036b54..f72dd8d6ac06f 100644 --- a/airflow-core/docs/core-concepts/dag-run.rst +++ b/airflow-core/docs/core-concepts/dag-run.rst @@ -50,24 +50,40 @@ Data Interval ------------- Each Dag run in Airflow has an assigned "data interval" that represents the time -range it operates in. For a Dag scheduled with ``@daily``, for example, each of -its data interval would start each day at midnight (00:00) and end at midnight -(24:00). +range it operates in. How that interval is defined depends on the Dag's +timetable. -A Dag run is usually scheduled *after* its associated data interval has ended, -to ensure the run is able to collect all the data within the time period. In -other words, a run covering the data period of 2020-01-01 generally does not -start to run until 2020-01-01 has ended, i.e. after 2020-01-02 00:00:00. +In Airflow 3, a Dag scheduled with a bare cron string such as ``@daily`` uses +:ref:`CronTriggerTimetable` by default (``[scheduler] create_cron_data_intervals`` +is ``False``). For that timetable, ``data_interval_start`` and +``data_interval_end`` are the same — the trigger time (for ``@daily``, midnight +each day). The run is created to execute *at* that time. + +If you need a contiguous full-day window instead — for example each run covering +from midnight to the next midnight — use a data-interval timetable such as +:ref:`CronDataIntervalTimetable`, or set ``[scheduler] create_cron_data_intervals`` +to ``True``. With that setup, a Dag run is usually scheduled *after* its +associated data interval has ended, so a run covering 2020-01-01 generally does +not start until after 2020-01-02 00:00:00. + +See :ref:`timetables comparison ` for a +side-by-side comparison, including how ``logical_date`` and ``run_id`` differ. All dates in Airflow are tied to the data interval concept in some way. The "logical date" (also called ``execution_date`` in Airflow versions prior to 2.2) -of a Dag run, for example, denotes the start of the data interval, not when the -Dag is actually executed. - -Similarly, since the ``start_date`` argument for the Dag and its tasks points to -the same logical date, it marks the start of *the Dag's first data interval*, not -when tasks in the Dag will start running. In other words, a Dag run will only be -scheduled one interval after ``start_date``. +of a Dag run is defined by the timetable: for both timetable kinds it is +``data_interval_start``. For the default (zero-width) trigger timetable that +equals the trigger time. With a non-zero ``interval=`` on a trigger timetable +it is ``trigger_time - interval``. For a data-interval timetable it is the +start of the contiguous window, not when the Dag is actually executed. + +Similarly, the ``start_date`` argument for the Dag and its tasks marks the +earliest *trigger time* (``run_after``) the scheduler will create, not when +tasks start running, and not necessarily the first logical date (see the +non-zero ``interval=`` case above). A run is only created once the timetable +reaches that bound. For a data-interval timetable, ``start_date`` is also the +start of the first data interval, so that first run does not execute until the +window closes, one schedule period after ``start_date``. .. tip:: @@ -104,10 +120,10 @@ Dag run fails. Catchup ------- -An Airflow Dag defined with a ``start_date``, possibly an ``end_date``, and a non-asset schedule, defines a series of intervals which the scheduler turns into individual Dag runs and executes. -By default, Dag runs that have not been run since the last data interval are not created by the scheduler upon activation of a Dag ( Airflow config ``scheduler.catchup_by_default=False``). The scheduler creates a Dag run only for the latest interval. +An Airflow Dag defined with a ``start_date``, possibly an ``end_date``, and a non-asset schedule, defines a series of scheduled run times which the scheduler turns into individual Dag runs and executes. +By default, missed scheduled run times between ``start_date`` and "now" are not backfilled when a Dag is activated (Airflow config ``scheduler.catchup_by_default=False``). The timetable instead selects the most recently applicable scheduled run time (for a trigger timetable, typically the latest cron tick that is not after "now" and not before ``start_date``). -If you set ``catchup=True`` in the Dag, the scheduler will kick off a Dag Run for any data interval that has not been run since the last data interval (or has been cleared). This concept is called Catchup. +If you set ``catchup=True`` in the Dag, the scheduler will kick off a Dag Run for any scheduled run time that has not been run since the last run (or has been cleared). This concept is called Catchup. If your Dag is not written to handle its catchup (i.e., not limited to the interval, but instead to ``Now`` for instance.), then you will want to turn catchup off, which is the default setting or can be done explicitly by setting ``catchup=False`` in the Dag definition, if the default config has been changed for your Airflow environment. @@ -138,20 +154,40 @@ then you will want to turn catchup off, which is the default setting or can be d ) In the example above, if the Dag is picked up by the scheduler daemon on -2016-01-02 at 6 AM, (or from the command line), a single Dag Run will be created -with a data between 2016-01-01 and 2016-01-02, and the next one will be created -just after midnight on the morning of 2016-01-03 with a data interval between -2016-01-02 and 2016-01-03. - -Be aware that using a ``datetime.timedelta`` object as schedule can lead to a different behavior. -In such a case, the single Dag Run created will cover data between 2016-01-01 06:00 and -2016-01-02 06:00 (one schedule interval ending now). For a more detailed description of the -differences between a cron and a delta based schedule, take a look at the -:ref:`timetables comparison ` - -If the ``dag.catchup`` value had been ``True`` instead, the scheduler would have created a Dag Run -for each completed interval between 2015-12-01 and 2016-01-02 (but not yet one for 2016-01-02, -as that interval hasn't completed) and the scheduler will execute them sequentially. +2016-01-02 at 6 AM (or from the command line), with the Airflow 3 default of +:ref:`CronTriggerTimetable` for ``@daily`` and ``catchup=False``, the scheduler +does **not** create runs for every midnight since ``start_date``. Instead it +creates a single Dag run for the most recent applicable tick — midnight on +**2016-01-02** — with ``data_interval_start`` and ``data_interval_end`` both +equal to that trigger time. Because that ``run_after`` is already in the past, +the run can start immediately. The following tick (midnight on 2016-01-03) is +only created once that schedule time is reached. + +If instead the Dag used a data-interval timetable (for example +:ref:`CronDataIntervalTimetable`, or ``[scheduler] create_cron_data_intervals=True``), +the scheduler would immediately create a run for the most recently completed +interval (2016-01-01 through 2016-01-02), and the next run would cover +2016-01-02 through 2016-01-03 after that interval ends. + +Be aware that using a ``datetime.timedelta`` object as ``schedule`` is not the +same as a cron string, even though Airflow 3 also defaults timedelta schedules +to a trigger timetable (:ref:`DeltaTriggerTimetable`, via +``[scheduler] create_delta_data_intervals``). A delta has no wall-clock boundary +to snap to, so with ``catchup=False`` the first run lands at pickup time +(**2016-01-02 06:00** in this example), with ``data_interval_start`` and +``data_interval_end`` both equal to that moment. If instead you enable the data-interval delta +timetable (``create_delta_data_intervals=True``), the first run covers one +schedule interval ending now (2016-01-01 06:00 through 2016-01-02 06:00). For a +more detailed description of the differences, see +:ref:`timetables comparison ` and +:ref:`cron vs delta data intervals `. + +If the ``dag.catchup`` value had been ``True`` instead, the scheduler would have +created a Dag Run for each scheduled run time between ``start_date`` and "now" +that had not yet run (or had been cleared). +With the default trigger timetable that means every midnight from 2015-12-01 +through 2016-01-02 inclusive. With a data-interval timetable, the still-open +interval that ends at the next midnight is not created yet. Catchup is also triggered when you turn off a Dag for a specified period and then re-enable it. diff --git a/airflow-core/docs/howto/deadline-alerts.rst b/airflow-core/docs/howto/deadline-alerts.rst index ac07047f34587..7eb9ec76cf2d0 100644 --- a/airflow-core/docs/howto/deadline-alerts.rst +++ b/airflow-core/docs/howto/deadline-alerts.rst @@ -431,8 +431,7 @@ choose a different time. from sqlalchemy.orm import Session - from airflow.sdk import DeadlineReference - from airflow.sdk.definitions.deadline import BaseDeadlineReference, deadline_reference + from airflow.sdk import BaseDeadlineReference, DeadlineReference, deadline_reference from airflow.sdk.timezone import datetime diff --git a/airflow-core/docs/howto/usage-cli.rst b/airflow-core/docs/howto/usage-cli.rst index d96cf5331f95f..21b37747e6888 100644 --- a/airflow-core/docs/howto/usage-cli.rst +++ b/airflow-core/docs/howto/usage-cli.rst @@ -223,6 +223,12 @@ The ``db clean`` command works by deleting from each table the records older tha You can optionally provide a list of tables to perform deletes on. If no list of tables is supplied, all tables will be included. +.. note:: + + Cleaning the ``asset_event`` table also drops any queued-but-not-yet-consumed asset events. This mainly + affects Dags waiting on a multi-asset condition, where a pending event can be purged before the condition + is met, meaning the Dag will not be triggered by it. + You can filter cleanup to specific DAGs using ``--dag-ids`` (comma-separated list), or exclude specific DAGs using ``--exclude-dag-ids`` (comma-separated list). These options allow you to target or avoid cleanup for particular DAGs. You can use the ``--dry-run`` option to print the row counts in the primary tables to be cleaned. diff --git a/airflow-core/docs/index.rst b/airflow-core/docs/index.rst index 4280171e42dc9..879ccb50e0871 100644 --- a/airflow-core/docs/index.rst +++ b/airflow-core/docs/index.rst @@ -19,8 +19,10 @@ What is Airflow®? ========================================= `Apache Airflow® `_ is an open-source platform for developing, scheduling, -and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows -connecting with virtually any technology. A web-based UI helps you visualize, manage, and debug your workflows. +and monitoring workflows, such as traditional time based or event-triggered batch-oriented data pipelines, machine learning, model training, +and agentic or LLM-based workloads. Airflow's extensible Python framework enables you to build workflows +connecting with virtually any technology, with a growing set of providers for orchestrating AI and agentic +tools alongside the rest of your pipeline. A web-based UI helps you visualize, manage, and debug your workflows. You can run Airflow in a variety of configurations — from a single process on your laptop to a distributed system capable of handling massive workloads. diff --git a/airflow-core/docs/migrations-ref.rst b/airflow-core/docs/migrations-ref.rst index dfc96be79019e..691720f6508a9 100644 --- a/airflow-core/docs/migrations-ref.rst +++ b/airflow-core/docs/migrations-ref.rst @@ -39,7 +39,12 @@ Here's the list of all the Database Migrations that are executed via when you ru +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | Revision ID | Revises ID | Airflow Version | Description | +=========================+==================+===================+==============================================================+ -| ``7a98f1b7dbd3`` (head) | ``c4e7a1f9b2d0`` | ``3.4.0`` | Add index on asset_event (asset_id, partition_key). | +| ``3c525f44bea8`` (head) | ``b2f1a9c7d4e0`` | ``3.4.0`` | Add indexes on serialized_dag and dag_code. | ++-------------------------+------------------+-------------------+--------------------------------------------------------------+ +| ``b2f1a9c7d4e0`` | ``7a98f1b7dbd3`` | ``3.4.0`` | Reference the asset event from asset_dag_run_queue (consume- | +| | | | by-reference). | ++-------------------------+------------------+-------------------+--------------------------------------------------------------+ +| ``7a98f1b7dbd3`` | ``c4e7a1f9b2d0`` | ``3.4.0`` | Add index on asset_event (asset_id, partition_key). | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ``c4e7a1f9b2d0`` | ``436dc127462c`` | ``3.4.0`` | Add index on asset.uri. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ diff --git a/airflow-core/docs/security/deprecated_permissions.rst b/airflow-core/docs/security/deprecated_permissions.rst index 35334c9c4306a..00014f1292520 100644 --- a/airflow-core/docs/security/deprecated_permissions.rst +++ b/airflow-core/docs/security/deprecated_permissions.rst @@ -51,10 +51,28 @@ replacement available from Airflow core: * ``airflow.security.permissions.ACTION_*`` --> ``airflow.api_fastapi.auth.managers.base_auth_manager.ResourceMethod`` * ``airflow.security.permissions.RESOURCE_*`` --> ``airflow.api_fastapi.auth.managers.models.resource_details`` -* ``DAG.access_control`` --> Dag-level permissions should be handled by the chosen Auth Manager's ``filter_authorized_dag_ids`` method. If you maintain a custom :doc:`/core-concepts/auth-manager/index` which relies on the deprecated module, it is recommended you refer to the ``SimpleAuthManager``'s `source code `_ as an example for how you might use the ``ResourceMethod`` and ``resource_details`` components. If you rely on custom role definitions based off the deprecated module, you should refer to the documentation of the auth manager your system uses. + +Migrating ``DAG.access_control`` +-------------------------------- + +Unlike the components above, ``DAG.access_control`` has no drop-in replacement. It was an input to the +FAB auth manager rather than an authorization mechanism of its own: at parse time its contents were +expanded into that manager's permission tables, and authorization then read those tables rather than +the Dag argument. + +.. warning:: + Airflow only reads ``DAG.access_control`` when the configured auth manager is the + :doc:`apache-airflow-providers-fab:auth-manager/index`. Under any other auth manager the argument is + parsed and serialized but never consulted, so it neither grants nor denies access. A deployment that + moves off the FAB auth manager and leaves ``access_control`` on its Dags stops enforcing those grants, + and does so without raising an error. + +To migrate, move the grants into the policy source your auth manager consults. The +:doc:`/core-concepts/auth-manager/index` guide covers how a custom auth manager should express +Dag-level access. diff --git a/airflow-core/docs/ui.rst b/airflow-core/docs/ui.rst index 23df5ce29b320..c65023fd7e157 100644 --- a/airflow-core/docs/ui.rst +++ b/airflow-core/docs/ui.rst @@ -229,6 +229,7 @@ The **Events** tab surfaces structured events related to the Dag, such as Dag tr .. image:: img/ui-light/dag_overview_events.png :alt: Dag Events Tab (Light Mode) + Code Tab '''''''' diff --git a/airflow-core/newsfragments/69821.bugfix.rst b/airflow-core/newsfragments/69821.bugfix.rst new file mode 100644 index 0000000000000..9dca1eb7d0cf9 --- /dev/null +++ b/airflow-core/newsfragments/69821.bugfix.rst @@ -0,0 +1 @@ +Deferrable tasks that fail via a trigger-emitted ``TaskFailedEvent`` now respect retries: if the task has retries remaining it goes ``up_for_retry`` and runs ``on_retry_callback``, instead of always failing terminally and running ``on_failure_callback``. diff --git a/airflow-core/newsfragments/70755.significant.rst b/airflow-core/newsfragments/70755.significant.rst new file mode 100644 index 0000000000000..f953fa49f40b4 --- /dev/null +++ b/airflow-core/newsfragments/70755.significant.rst @@ -0,0 +1,29 @@ +Team scoped values of options registered as sensitive are now hidden + +Configuration options are registered as sensitive under their base section, so until now only the +base spelling of an option was masked. A team scoped override -- set in a ``[=
]`` +config file section, or through an ``AIRFLOW_____
__`` environment variable -- +was not recognised as the same option and was returned in full. + +Sensitivity is now decided after resolving the team scoped spelling back to the base option, so a +team scoped value is masked exactly as the base value already was. + +**Behaviour changes:** + +- ``AirflowConfigParser.as_dict(display_sensitive=False)``, ``GET /config``, + ``GET /config/section/{section}/option/{option}`` and ``airflow config list`` now return + ``< hidden >`` for a team scoped value of an option registered as sensitive. Deployments that + read a team's real value through any of these will now receive the mask; use + ``display_sensitive=True`` where a real value is required and appropriate. +- Team scoped ``_cmd`` and ``_secret`` entries are replaced with ``< hidden >`` in place, rather + than being resolved into their value and removed as they are in a base section. Resolving them + is not supported for a team, so the command string or secret path is no longer shown either. +- Non team configuration is unaffected, and ``display_sensitive=True`` continues to return real + values. + +**New public helpers on the shared configuration parser** (additive; no signatures changed): + +- ``team_section_name(team_name, section)`` builds the team scoped config file section name. +- ``base_section_name(section)`` recovers the base section from a possibly team scoped one. +- ``AirflowConfigParser.is_sensitive_option(section, key)`` reports whether an option is + registered as sensitive under any of its spellings. diff --git a/airflow-core/newsfragments/70791.significant.rst b/airflow-core/newsfragments/70791.significant.rst deleted file mode 100644 index 827f22bc68f68..0000000000000 --- a/airflow-core/newsfragments/70791.significant.rst +++ /dev/null @@ -1,35 +0,0 @@ -pandas is limited to ``< 3``; DataFrame XComs are refused on pandas 3 - -A ``pandas.DataFrame`` does not survive an XCom round trip under pandas 3 the way it does -under pandas 2, so this version of Airflow constrains pandas to ``< 3`` and refuses DataFrame -XComs at runtime when a newer pandas is installed anyway. - -Two things change under pandas 3. Its public classes are exposed from the ``pandas`` -namespace, so a DataFrame qualifies as ``pandas.DataFrame`` rather than -``pandas.core.frame.DataFrame`` and the XCom serializer registry -- keyed on the latter -- -never dispatches to the pandas serializer. And the dtypes differ: an ``object`` column reads -back as ``str``, and missing values as ``nan`` rather than ``None``, so the *reader's* pandas -version decides what a task receives rather than the writer's. - -Because pandas is an optional dependency, the ``< 3`` constraint alone cannot be relied on -- -a deployment may install pandas 3 directly, or pull it in through another package. It is -therefore backed by a runtime check. - -**Behaviour changes:** - -- With pandas 3 or newer installed, pushing **or** pulling a ``DataFrame`` through XCom now - raises ``RuntimeError`` naming the installed pandas version and the supported range. - Reads are refused as well as writes, because a payload written under pandas 2 reads back - with pandas 3 dtypes and would otherwise hand the task different data than was pushed, - with no signal. -- Previously the same situation produced an opaque - ``TypeError: cannot serialize object of type `` from the - serializer registry, with nothing identifying pandas as the cause. -- Deployments that pass DataFrames through XCom must pin ``pandas < 3`` in the environment - that runs their tasks. Deployments that do not use DataFrame XComs are unaffected at - runtime. -- Nothing changes for pandas 2 users. - -**This defers pandas 3 support rather than dropping it.** Moving from pandas 2 to pandas 3 -is a deliberate migration for users to make, given the dtype changes it brings to data that -has already been written. Support for pandas 3 will be enabled in a future Airflow release. diff --git a/airflow-core/newsfragments/70972.significant.rst b/airflow-core/newsfragments/70972.significant.rst new file mode 100644 index 0000000000000..bae78066f12c6 --- /dev/null +++ b/airflow-core/newsfragments/70972.significant.rst @@ -0,0 +1,13 @@ +Changes to how pending asset events trigger Dag runs + +Deleting asset event records with ``airflow db clean`` now also removes any +asset events that are still pending for a downstream Dag. This mainly affects +Dags waiting on a multi-asset condition, where a pending event can be purged +before the condition is met. Deleting an asset event now consistently means it +will not trigger a run; previously such a pending trigger could still fire after +the event had been purged. + +Relatedly, an asset event that was queued for a Dag while it was scheduled on +that asset is still used to trigger the run, and appears among the run's +triggering events, even if the asset is later removed from the Dag's schedule. +Previously, they were silently dropped. diff --git a/airflow-core/newsfragments/71103.significant.rst b/airflow-core/newsfragments/71103.significant.rst new file mode 100644 index 0000000000000..9dd43544b0dfb --- /dev/null +++ b/airflow-core/newsfragments/71103.significant.rst @@ -0,0 +1,33 @@ +pandas 3 changes how DataFrame XComs are stored and read back + +pandas 3 exposes its public classes from the ``pandas`` namespace, so a DataFrame is qualified as +``pandas.DataFrame`` instead of ``pandas.core.frame.DataFrame``. XComs record that name alongside the +serialized value, so the name written into the metadata database depends on the pandas version of the +component that pushed the value. Airflow registers both names, and a DataFrame written by either +pandas version can be read by either — no configuration change is needed, and existing XComs stay +readable. + +What you should do: + +* **Roll this Airflow version out to every component before pandas 3 reaches any of them** — workers + in particular. A component that predates this change cannot read a DataFrame XCom written under + pandas 3, and fails the pull with: + + .. code-block:: text + + ImportError: pandas.DataFrame was not found in allow list for deserialization imports. + To allow it, add it to allowed_deserialization_classes in the configuration + + The message points at configuration, but the allow list is not the cause and changing it does not + help. The rows are not corrupt: they become readable again as soon as the reader is upgraded. + +* **Treat a downgrade as a one-way door for those XComs.** Rolling back to an Airflow version without + this change strands any DataFrame XCom written while on pandas 3, with the same error, until you + roll forward again. + +* **Review Dags that inspect the dtypes of a pulled DataFrame.** The pandas version of the *reader* + determines what a pulled DataFrame looks like, not the version that wrote it. Under pandas 3, a + column of strings comes back with the ``str`` dtype rather than ``object``, and its missing values + come back as ``nan`` rather than ``None``. Values are unchanged, but downstream code that branches + on ``dtype == "object"``, checks cells with ``is None``, or compares against a reference frame with + ``DataFrame.equals()`` can behave differently after the upgrade. diff --git a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/package.json b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/package.json index c214469185c7c..979c5e52f9241 100644 --- a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/package.json +++ b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/package.json @@ -22,13 +22,13 @@ "@hey-api/client-axios": "^0.9.1", "@hey-api/openapi-ts": "^0.99.0", "@tanstack/react-query": "^5.101.4", - "axios": "^1.18.1", + "axios": "^1.19.0", "next-themes": "^0.4.6", "react": "^19.2.8", "react-cookie": "^8.1.2", "react-dom": "^19.2.8", - "react-hook-form": "^7.83.0", - "react-router-dom": "^7.18.1" + "react-hook-form": "^7.84.0", + "react-router-dom": "^7.18.2" }, "devDependencies": { "@7nohe/openapi-react-query-codegen": "^2.2.0", @@ -38,12 +38,12 @@ "@testing-library/jest-dom": "^7.0.0", "@testing-library/react": "^16.3.2", "@trivago/prettier-plugin-sort-imports": "^6.0.2", - "@types/react": "^19.2.17", - "@types/react-dom": "^19.2.3", + "@types/react": "^19.2.18", + "@types/react-dom": "^19.2.4", "@typescript-eslint/eslint-plugin": "8.65.0", "@typescript-eslint/parser": "8.65.0", "@typescript-eslint/utils": "^8.65.0", - "@vitejs/plugin-react-swc": "^4.3.2", + "@vitejs/plugin-react-swc": "^4.3.3", "@vitest/coverage-v8": "^4.1.10", "eslint": "^10.8.0", "eslint-config-prettier": "^10.1.8", @@ -59,7 +59,7 @@ "ts-morph": "^28.0.0", "typescript": "~6.0.3", "typescript-eslint": "^8.65.0", - "vite": "^8.1.5", + "vite": "^8.2.0", "vite-plugin-css-injected-by-js": "^5.0.2", "vitest": "^4.1.10" }, diff --git a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/pnpm-lock.yaml b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/pnpm-lock.yaml index 0d3c10d313110..c29d206c5505f 100644 --- a/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/pnpm-lock.yaml +++ b/airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui/pnpm-lock.yaml @@ -35,10 +35,10 @@ importers: dependencies: '@chakra-ui/react': specifier: ^3.36.1 - version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@hey-api/client-axios': specifier: ^0.9.1 - version: 0.9.1(@hey-api/openapi-ts@0.99.0(magicast@0.5.3)(typescript@6.0.3))(axios@1.18.1) + version: 0.9.1(@hey-api/openapi-ts@0.99.0(magicast@0.5.3)(typescript@6.0.3))(axios@1.19.0) '@hey-api/openapi-ts': specifier: ^0.99.0 version: 0.99.0(magicast@0.5.3)(typescript@6.0.3) @@ -46,8 +46,8 @@ importers: specifier: ^5.101.4 version: 5.101.4(react@19.2.8) axios: - specifier: ^1.18.1 - version: 1.18.1 + specifier: ^1.19.0 + version: 1.19.0 next-themes: specifier: ^0.4.6 version: 0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -56,16 +56,16 @@ importers: version: 19.2.8 react-cookie: specifier: ^8.1.2 - version: 8.1.2(@types/react@19.2.17)(react@19.2.8) + version: 8.1.2(@types/react@19.2.18)(react@19.2.8) react-dom: specifier: ^19.2.8 version: 19.2.8(react@19.2.8) react-hook-form: - specifier: ^7.83.0 - version: 7.83.0(react@19.2.8) + specifier: ^7.84.0 + version: 7.84.0(react@19.2.8) react-router-dom: - specifier: ^7.18.1 - version: 7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + specifier: ^7.18.2 + version: 7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) devDependencies: '@7nohe/openapi-react-query-codegen': specifier: ^2.2.0 @@ -84,16 +84,16 @@ importers: version: 7.0.0(@testing-library/dom@10.4.0) '@testing-library/react': specifier: ^16.3.2 - version: 16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@trivago/prettier-plugin-sort-imports': specifier: ^6.0.2 version: 6.0.2(prettier@3.9.6) '@types/react': - specifier: ^19.2.17 - version: 19.2.17 + specifier: ^19.2.18 + version: 19.2.18 '@types/react-dom': - specifier: ^19.2.3 - version: 19.2.3(@types/react@19.2.17) + specifier: ^19.2.4 + version: 19.2.4(@types/react@19.2.18) '@typescript-eslint/eslint-plugin': specifier: 8.65.0 version: 8.65.0(@typescript-eslint/parser@8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3))(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3) @@ -104,8 +104,8 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3) '@vitejs/plugin-react-swc': - specifier: ^4.3.2 - version: 4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) + specifier: ^4.3.3 + version: 4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/coverage-v8': specifier: ^4.1.10 version: 4.1.10(vitest@4.1.10) @@ -152,14 +152,14 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3) vite: - specifier: ^8.1.5 - version: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) + specifier: ^8.2.0 + version: 8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) vite-plugin-css-injected-by-js: specifier: ^5.0.2 - version: 5.0.2(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) + version: 5.0.2(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) vitest: specifier: ^4.1.10 - version: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) + version: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) packages: @@ -309,15 +309,6 @@ packages: react: '>=18' react-dom: '>=18' - '@emnapi/core@1.11.1': - resolution: {integrity: sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==} - - '@emnapi/runtime@1.11.1': - resolution: {integrity: sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==} - - '@emnapi/wasi-threads@1.2.2': - resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} - '@emotion/babel-plugin@11.13.5': resolution: {integrity: sha512-pxHCpT2ex+0q+HH91/zsdHkw/lXd468DIN2zvfvLtPKLLMo6gQj7oLObq8PhkrxOZb/gGCq03S3Z7PDhS8pduQ==} @@ -510,14 +501,8 @@ packages: resolution: {integrity: sha512-9I2Zn6+NJLfaGoz9jN3lpwDgAYvfGeNYdbAIjJOqzs4Tpc+VU3Jqq4IofSUBKajiDS8k9fZIg18/z13mpk1bsA==} engines: {node: '>=8'} - '@napi-rs/wasm-runtime@1.1.6': - resolution: {integrity: sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==} - peerDependencies: - '@emnapi/core': ^1.7.1 - '@emnapi/runtime': ^1.7.1 - - '@oxc-project/types@0.139.0': - resolution: {integrity: sha512-r9gHphtCs+1M7J0pw6Sn/hh/Wpa/iQrOOkrNAlVLF/gHq+/CJmHIWKKUUhdWjcD6CIa8idarspCsASiXCXvFUw==} + '@oxc-project/types@0.143.0': + resolution: {integrity: sha512-u6JZdLBTLotrNC9Vd6vPssINdzcCzleKAH6EJKImQb7GtYvX5keN2dxkoK44stCc4tffE6QQRtZTXVSzsLUlWA==} '@pandacss/is-valid-prop@1.11.5': resolution: {integrity: sha512-8m1/gG0wSRMa1QypCkWzeKUeMKY7k14nxsJCHs7Z1g1fI9I7WsLjW/VI58bLZdJxAxtoreMtm6TfIITda8P4zQ==} @@ -527,97 +512,92 @@ packages: resolution: {integrity: sha512-SEeaJLb3qBNF/OaXnaR1NmmBbFYk1zC0ZH/52fATcRPLFg/p791YrcyFFy44Bo9sLaGuSuLp5Q6axbb/O+v/RA==} engines: {node: ^14.18.0 || >=16.0.0} - '@rolldown/binding-android-arm64@1.1.5': - resolution: {integrity: sha512-lZg8fqIv2v7FF237bwMgzGZEJvGL79/s5knJ/i6FmsGF4XXlzccZ4jb+TrFIxtSSxFtIpdsgrPZeMk1I9AFcyQ==} + '@rolldown/binding-android-arm64@1.2.3': + resolution: {integrity: sha512-zrJtHDcaZJ1Fp7xf4hNl+7seH9Cn/N5TwLYkhgXREtBwAd/jaqW3uqeHxpDugJLVICWg4eW44kOQEGJ1r6jCGw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@rolldown/binding-darwin-arm64@1.1.5': - resolution: {integrity: sha512-51Bnx9pNiMRKSUNtBfySkNJ9vMU9Hh3I1ozDd6gyPPYzaXCfnptUcEZxXGYFn+ul2dtcMUiqGR1Yai2K10uoTw==} + '@rolldown/binding-darwin-arm64@1.2.3': + resolution: {integrity: sha512-ieIiibVCp0tX7TLu2cafoNPv8wJyYi01ekXpbf8q2j7F4rGAhhXb/eQh7ge9DRBY78GwmRQtvjZDux7EDbA8kA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@rolldown/binding-darwin-x64@1.1.5': - resolution: {integrity: sha512-Tm+gbfC0aHu1tBA/JvKQh32S0K6YgCHkiAF4/W6xX0K0RmNuc94VeK419dJoE65R5aRxmo+noZQSWrAMF6yb6g==} + '@rolldown/binding-darwin-x64@1.2.3': + resolution: {integrity: sha512-Zh9tCon19eDXJoihx0rqKhMUlMYqzwj3aPsSuHmI4RWZh62dWUL+DJN4C5YQya5TcQBJU/Fe8+rY0jhXTQITqA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@rolldown/binding-freebsd-x64@1.1.5': - resolution: {integrity: sha512-JMzDKCCXq93YccG5gz3hvOs1oXRKAf0XYpfOS88e+wZrC8Iugj6j68867vrYZkvpDDpKn/KoKORThmchMpF6TA==} + '@rolldown/binding-freebsd-x64@1.2.3': + resolution: {integrity: sha512-nGbJWewA1wrXXZiQhjAT5rhibGfns5ZNkDVqxsO6zJ3f3YvpoDNNmGMSbbhLuXKjNScaBJVOAboztAWVespQMg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': - resolution: {integrity: sha512-uML21j2K5TfPGutKxub+M+nLjZIrWjXQ5Grx4lCe/nimTj9B4L63zHpjXLl4y0L3mcm2htEQIb06oCG/szerNw==} + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': + resolution: {integrity: sha512-QNniJr5Kml0kDEB98jiDOJjXNroxIIi0IXIbdYzY26Xt1pVbeP62+KnoIZLwirOymX/0jDk/2gI/bNUv7A7OIw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@rolldown/binding-linux-arm64-gnu@1.1.5': - resolution: {integrity: sha512-navSiuTMogvnQoZoM/v+l3ZWo50/NTwSHSzheABx/RCnmUPaKwq9qSo4Br2OYRs21+Fz8uFqITZM3H4opOB0/Q==} + '@rolldown/binding-linux-arm64-gnu@1.2.3': + resolution: {integrity: sha512-TkqEAcmmvH3I/q4114NB4RVt6241Dao48pF45uLcFGrwAaIn0iITgTAKP/dLjbN0R4buJjGb91+UHSoFmpgIWw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-arm64-musl@1.1.5': - resolution: {integrity: sha512-lAryqH7IteztmCXQXk0etKj4wBQ7Gx5S6LjKhsgp9zb8I5bsuvU/2llH1hDQcjsFeqIsovMVN339/8pUDDBXxA==} + '@rolldown/binding-linux-arm64-musl@1.2.3': + resolution: {integrity: sha512-NHqjnxpsndf4MPymxteFAWHHfkTL8HjWh1KB7z23ofZ6QO2euONuxDXjat69dKZRALnGypg8k8SsK8vZJoXv1Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@rolldown/binding-linux-ppc64-gnu@1.1.5': - resolution: {integrity: sha512-fsK/sNBnxzBlL4O1JNrZakVQxPspqpED5dLtNsZS9oOKmtSpdNIzxH2kkol5HYTWJN47sE20ztMJPxfZ89qGOg==} + '@rolldown/binding-linux-ppc64-gnu@1.2.3': + resolution: {integrity: sha512-6tbrbwfz5GB9DQ4Jwo6hy9v+vR31xZlvzZ6n5Xut6Hhx5PvrA9q/HsK8KMaYQp063iqZGXwNvZtYNLD7EM/x0w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-s390x-gnu@1.1.5': - resolution: {integrity: sha512-gLYb4BIadlfTOYT5gO503n8zQjXflgzpD0FcyKh0Mzx3rqCZKnHoJWV9xe1KXUJ5lx2JfcSHr/mhzS0PC/McAA==} + '@rolldown/binding-linux-s390x-gnu@1.2.3': + resolution: {integrity: sha512-oyuXxXmoZHjXC917IAPFAAv4wWAa0cM9afk8nx1+9/jNNOX1uPf8yDA6p7G0RypOfw/X0PQt5IfoquY1um+zSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@rolldown/binding-linux-x64-gnu@1.1.5': - resolution: {integrity: sha512-FjcpEKUyJygHgs1o50VYNvkt5+7Le/VEdYt0AkRpkL33MnyQfwr8l5mXwMmfmTbyMPr5vJLC+8/Gd9gXnwU1QQ==} + '@rolldown/binding-linux-x64-gnu@1.2.3': + resolution: {integrity: sha512-TytMwF2KVGqP2tgd0I1OY0PAv78dZRAYcF5ssDzjM34SUXCED3uXvSd5+lHoC0bTD6eEdFz7LdQNCO1y0oVk9w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-x64-musl@1.1.5': - resolution: {integrity: sha512-Me+PfPI2TMeOQk0gYWfLQZtTktrmzbr8cDboqX83XKc7UrgAi55gF+2dUkWdxd19n55Essp2yeca+O9N5rBxHg==} + '@rolldown/binding-linux-x64-musl@1.2.3': + resolution: {integrity: sha512-/E9m3qstrJFVPoULV25mVQblSNExY2+kBsYe4sy0Tn0yOOgJ8wZbZt3KnRbF/XeU2Gl1STKUQnDNTqhIE5MD4A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@rolldown/binding-openharmony-arm64@1.1.5': - resolution: {integrity: sha512-yc5WrLzXks6zCQfn9Oxr8pORKyl/pF+QjHmW/Qx3qu0oyrrNC+y2JLTU1E2rcWYAmzlnqngWXHQjy51VzW70Vw==} + '@rolldown/binding-openharmony-arm64@1.2.3': + resolution: {integrity: sha512-Kr0OcsoQI816i6HOl3vFHpd1K0eZyh76zgfj4c1nTyaTsd5r2Mj1lwM4R90y/qaCfmTn9eHy0SKwi98eitRxug==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@rolldown/binding-wasm32-wasi@1.1.5': - resolution: {integrity: sha512-VbQGPX2b4r48TAMIM2cjgluIM1HYutm4pcTEJsle7iEP7sB1dFqtPLBVbdLAZCxy1txCcPxf4QFf4v8uvltPqA==} - engines: {node: ^20.19.0 || >=22.12.0} - cpu: [wasm32] - - '@rolldown/binding-win32-arm64-msvc@1.1.5': - resolution: {integrity: sha512-gHv82k63z4qpV5+Q1y/12KrK0ltWBukVDI8nZcbT7Tt/ZlOIVwppazneq0F93oDxTo3IgAMEDIoQh3E2n6mVsw==} + '@rolldown/binding-win32-arm64-msvc@1.2.3': + resolution: {integrity: sha512-hOtMwTqnME+/gJcH/PCZ0wn0zPUjiWOgkHpxbSJpfGKMezHltx1S7/k1SitzVa7Ww2cqrDDaFbZEhcJZO8o+Jw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@rolldown/binding-win32-x64-msvc@1.1.5': - resolution: {integrity: sha512-tTZuDBPw85tEN5PQi1pnEBzDy0Z49HtScLAbD5t6hyeU92A95pRWaSMw1GZZi/RwgSgUIl0xrSlXIT/9QzvYSA==} + '@rolldown/binding-win32-x64-msvc@1.2.3': + resolution: {integrity: sha512-ekcqMMkI2PlhYnfzQnB/cEdYUVVJViWvoUyLrbzgDoi3Snfc1mVBwdnc306ufA5ejy8JSPjT2RlW1nQSjW7efg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -634,86 +614,86 @@ packages: peerDependencies: eslint: ^9.0.0 || ^10.0.0 - '@swc/core-darwin-arm64@1.15.46': - resolution: {integrity: sha512-IsISIT22EfktVJrlvIpnAxG2u/A9aob9l99HMlx80x72WlFmFPk1V3UhkEzx86eJP8hw049KTFv/RISho2cq2Q==} + '@swc/core-darwin-arm64@1.15.47': + resolution: {integrity: sha512-GsoMtan3ojGGMGFbl31mmRu5ctZ56re8grGE8mO/OHJ8O+JRkzod02fe7X6ZQ8JvamA3imkEkx/h3u+vsOgPgA==} engines: {node: '>=10'} cpu: [arm64] os: [darwin] - '@swc/core-darwin-x64@1.15.46': - resolution: {integrity: sha512-4Tj4ppVIPCmUMpmGFiGtyEriwLyJ+yi/US4WfBrP/ok8COGddDZXLEzQETnKyK46mjvr1v0jevrS23zjoff7vA==} + '@swc/core-darwin-x64@1.15.47': + resolution: {integrity: sha512-leTi7Rx3KF4zcC637iqWgk9SoV8VXAD8ppQYXsep63px5A/UftOcxLN1pmr8Z1si/YvX90ompP/rHgpYkgwXWg==} engines: {node: '>=10'} cpu: [x64] os: [darwin] - '@swc/core-linux-arm-gnueabihf@1.15.46': - resolution: {integrity: sha512-i8tUGnNjyOgMmfmgFSg4aeJLQoFyfpIHK5FjpQAwpRyQIqEUB2w1e8zIDQzY1WhOxx8NoS1S5iUL813Un4Sf5A==} + '@swc/core-linux-arm-gnueabihf@1.15.47': + resolution: {integrity: sha512-hBqHuoWKKIsKmDBn9qVeWqj5GWZhtlcczVaqQmNRXsDfq+voR5CxKRfamA367QjJXtceYuliLFfEL8QsskRM2g==} engines: {node: '>=10'} cpu: [arm] os: [linux] - '@swc/core-linux-arm64-gnu@1.15.46': - resolution: {integrity: sha512-c0OnhqzdhfOvv6qhNCcByepB+sNYOGZyhtr2Qa6ZCHvAWTYhSRw4j/u92Stue9PbZ/6q74b9nHzi76+kVzqQHQ==} + '@swc/core-linux-arm64-gnu@1.15.47': + resolution: {integrity: sha512-TBxvRz+B4K205TWHHZxWVxkC2RFNP/Mz3PNcECBos5PsKwxjg3QSJzdoebr0VCf0Bfh8HOPldKxAP/8XkFe9gA==} engines: {node: '>=10'} cpu: [arm64] os: [linux] libc: [glibc] - '@swc/core-linux-arm64-musl@1.15.46': - resolution: {integrity: sha512-imyRpNEcUzFQFV2LE4jL68ErvmKEuZCbvZru77iQREunJ+bR4i658cupTgtG1mLYM3F1Tzy3Sb9xYb02KghWTg==} + '@swc/core-linux-arm64-musl@1.15.47': + resolution: {integrity: sha512-3Yu3Uq/VgytqsPjTMbkPU1ExADytbdWbruJYhA584E9jrpE2Ki+R6VVPoZCeAVk1Cb7QxcRTgblw6bSa6a/R+w==} engines: {node: '>=10'} cpu: [arm64] os: [linux] libc: [musl] - '@swc/core-linux-ppc64-gnu@1.15.46': - resolution: {integrity: sha512-ctEfcl/HcUeomK33cbySiHZm98GEDIxTm1EkpBsYCiHxElYBzvTXVeuQT2YwbUXn9XCrjiw4ipyUNk33k26qRg==} + '@swc/core-linux-ppc64-gnu@1.15.47': + resolution: {integrity: sha512-wfdMi5IaOaNtmh2/6geRoxIdNfqylUZFdtzTKS655y1axWfIWyx7As74vv0wVdjeCIZ3WmCI9odDd4rUttXOSQ==} engines: {node: '>=10'} cpu: [ppc64] os: [linux] libc: [glibc] - '@swc/core-linux-s390x-gnu@1.15.46': - resolution: {integrity: sha512-DxlMdnt84TtRVTv7WL/thWyz9+QU8QZNNoAP9rrk0P68LziuhfePp8MjQ44zIprpTHTsEwyziIuGUUN5iSC1bQ==} + '@swc/core-linux-s390x-gnu@1.15.47': + resolution: {integrity: sha512-3hHYBY0yx8Ez7GMRrkhXHQzMdR5IZA6Wq5Ee4svlgwvSECLpnAJ9+0AimEGUFDvuLwE7nV/2+PYe8+Nm4rvNcQ==} engines: {node: '>=10'} cpu: [s390x] os: [linux] libc: [glibc] - '@swc/core-linux-x64-gnu@1.15.46': - resolution: {integrity: sha512-SKxI7J6t90XPl8hRUqtJi9NfGdunN/E/vZMc7Bc0figeRdOPDBT+Tm8g7cx9xM0T0mewh2l+8dewa3Am27/P+A==} + '@swc/core-linux-x64-gnu@1.15.47': + resolution: {integrity: sha512-TjfhjgP/jGCfFHYC3JQPhJA1HwErbIJ9JfREDc1KNkvY6P0LodCgKVIlQ5deeTbkG7ih3bF5PHJLuLpaZjdRyQ==} engines: {node: '>=10'} cpu: [x64] os: [linux] libc: [glibc] - '@swc/core-linux-x64-musl@1.15.46': - resolution: {integrity: sha512-qj9T6B7bosI0VEsrWOVXZN1OXxS8Tp63ywyrLxNdOycnUtLdkgYcoBsN5y8ImnDDsnwrEWZOy1e+J4xSe7mA3Q==} + '@swc/core-linux-x64-musl@1.15.47': + resolution: {integrity: sha512-CQpS8Ge/avfjZd0UEwG/sds83Uu32deQXcV1Jo3jD0mmvQQqtYAjpsDZXugmheeAwmt+YIuoVtVHro8LMYHqsQ==} engines: {node: '>=10'} cpu: [x64] os: [linux] libc: [musl] - '@swc/core-win32-arm64-msvc@1.15.46': - resolution: {integrity: sha512-8p7l4c3LU+eA5g9Et1JPhNeMC1oQwXTGU+uah8DPIBX7YXzqswvaBtyKVmXefVGi/DJU1x3YJsc3mbAp9aWzSQ==} + '@swc/core-win32-arm64-msvc@1.15.47': + resolution: {integrity: sha512-0W8IKHsUTYiT7G2RqtOoVWk+89yzZikIiDUb/sCK6BmQDBhN91hQSfyUtW12jhEWLzYgcfmisfsZrmZE+84U1A==} engines: {node: '>=10'} cpu: [arm64] os: [win32] - '@swc/core-win32-ia32-msvc@1.15.46': - resolution: {integrity: sha512-tUEnfr3Bn9u6FOjUb3PN9p+09qZC2j+wNDLKHzXXZn22rqGcUqR/ohCRSS+nG9B9+X+U+3FewNEHJkTmdIvMjQ==} + '@swc/core-win32-ia32-msvc@1.15.47': + resolution: {integrity: sha512-ZIp49d2Z4/ka2jO9otOg4hDvTdPmp86kVOgS2M5FCPI7eKKZ1W0boxWn+8XeZrfERtFGW0AlMRm4JhlJa7l3NA==} engines: {node: '>=10'} cpu: [ia32] os: [win32] - '@swc/core-win32-x64-msvc@1.15.46': - resolution: {integrity: sha512-Vux7UDzBJYQggSuPfcl2w9iu+IJpgpRCxHzgCaVkELnAXAE4XZMOTX9HNcaNiwfeIDqdu2rkr69RuDm6wY8neA==} + '@swc/core-win32-x64-msvc@1.15.47': + resolution: {integrity: sha512-2h8Iek95vnixkBRCo+H8p09+Q5ll2NgSMFrWTy0iKt7+/t+8/T5mBpiT6c0ZxSS7wcWjwZ9sGZkK70tTSYHdDw==} engines: {node: '>=10'} cpu: [x64] os: [win32] - '@swc/core@1.15.46': - resolution: {integrity: sha512-Ri3em2mBpq3h2zSPliCYl63otDGqek8PPEfv2nWgRQEbZ/VBCNyypVTVQ6cEbTCXBhy+WE2T3fQb08moIyuYaw==} + '@swc/core@1.15.47': + resolution: {integrity: sha512-FbsO5JcfOjfH38W/rohBRBweJeERsAuIP4f377lmkmxTcq9exjtx4SkRuZY5CdfhR2CBVwDIJegBpJDffwNsOg==} engines: {node: '>=10'} peerDependencies: '@swc/helpers': '>=0.5.17' @@ -727,8 +707,8 @@ packages: '@swc/helpers@0.5.23': resolution: {integrity: sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==} - '@swc/types@0.1.27': - resolution: {integrity: sha512-K6h3iUlqeM946U4sXFYeahefR1YBbXJvko+hv8WS8/0BNJ4OHiHRywMnQUJCqkR7Y9+hqQ1TvEpiKqUhz7NEFg==} + '@swc/types@0.1.28': + resolution: {integrity: sha512-V6Mnml8v09QALx6K0elJ7o9K/MkVDtW3t6L+7Ou/JcWtb3xwId2AH4FeOceySd2JaO87IMw4+6vSZxLm34LPbw==} '@tanstack/query-core@5.101.4': resolution: {integrity: sha512-gNwcvOJcRbLWPOLG/2OBm+zM+Yv+MKsXKEOWC57USuZDEsI71hEErQsiEGx5wX9rzWWkfwM0fVSPoiIFSsxfiw==} @@ -785,9 +765,6 @@ packages: '@ts-morph/common@0.29.0': resolution: {integrity: sha512-35oUmphHbJvQ/+UTwFNme/t2p3FoKiGJ5auTjjpNTop2dyREspirjMy82PLSC1pnDJ8ah1GU98hwpVt64YXQsg==} - '@tybys/wasm-util@0.10.3': - resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} - '@types/aria-query@5.0.4': resolution: {integrity: sha512-rfT93uj5s0PRL7EzccGMs3brplhcrghnDoV26NqKhCAS1hVo+WdNsPvE/yb6ilfr5hi2MEk6d5EWJTKdxg8jVw==} @@ -817,13 +794,13 @@ packages: '@types/parse-json@4.0.2': resolution: {integrity: sha512-dISoDXWWQwUquiKsyZ4Ng+HX2KsPL7LyHKHQwgGFEA3IaKac4Obd+h2a/a6waisAoepJlBcx9paWqjA8/HVjCw==} - '@types/react-dom@19.2.3': - resolution: {integrity: sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==} + '@types/react-dom@19.2.4': + resolution: {integrity: sha512-Bsc+QHgp+P/F02XDzNCY9jnZNCUuLki36KT7VKrTXXLdHf+vHMNZnW1rVu5DNW/rCK+fya3DATySbLM4yhtKUw==} peerDependencies: '@types/react': ^19.2.0 - '@types/react@19.2.17': - resolution: {integrity: sha512-MXfmqaVPEVgkBT/aY0aGCkRWWtByiYQXo3xdQ8r5RzuFrPiRn8Gar2tQdXSUQ2GKV3bkXckek89V8wQBY2Q/Aw==} + '@types/react@19.2.18': + resolution: {integrity: sha512-AnzbBERsrLKtk2XSfTbYRLjQPdy116Sty4q+T+Bp3IC4l6jNBvreVPAHmpq9qhXQM7CXZPjLVmGMw9sy+hxQ3w==} '@types/whatwg-mimetype@3.0.2': resolution: {integrity: sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA==} @@ -894,8 +871,8 @@ packages: resolution: {integrity: sha512-8C71BQkGjiMmXtop7pHVJu1l2NNShFdkCyD6a2ezzs5vU/L3LRtb69EtcteFwz0mYMPzIgOw0n6OV4VBUWZd7A==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} - '@vitejs/plugin-react-swc@4.3.2': - resolution: {integrity: sha512-MlSlmSAYbYwPjGa+CjOAqwij09iPYAQDHwx8osVkwoDamCxXlg+avpkC65Ysv+L/uqUWvRTsnJCKPjVhZih/sA==} + '@vitejs/plugin-react-swc@4.3.3': + resolution: {integrity: sha512-bti8ZAcvz4Lh6/e4Uk2k3aa1TiUXbbMsahuqOHvd3MveFTkKDZOA6wQVkpj7J/+tepX/wGfe+lsGh/t24HTXMQ==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: vite: '>=7.3.2' @@ -1273,8 +1250,8 @@ packages: resolution: {integrity: sha512-Xm7bpRXnDSX2YE2YFfBk2FnF0ep6tmG7xPh8iHee8MIcrgq762Nkce856dYtJYLkuIoYZvGfTs/PbZhideTcEg==} engines: {node: '>=4'} - axios@1.18.1: - resolution: {integrity: sha512-3nTvFlvpn9Zu/RkHUqtc7/+al4UpRW5az71ap5zccp6e8RAYEzhMTecX8Dz1wWDYrPpUoB1HAQEGEAEvUr7S9g==} + axios@1.19.0: + resolution: {integrity: sha512-ht/iuYZXEjFxLH/Hkezgd7m6JKlHHXEUSneaDz8uZe1Gj5QZtCnpyDsckvAiEnT89OEbCLmnte4R4sn7P0EKFw==} axobject-query@4.1.0: resolution: {integrity: sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==} @@ -2237,8 +2214,8 @@ packages: ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} - nanoid@3.3.16: - resolution: {integrity: sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==} + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true @@ -2385,8 +2362,8 @@ packages: resolution: {integrity: sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg==} engines: {node: '>= 0.4'} - postcss@8.5.23: - resolution: {integrity: sha512-g50586zr4bZmwFiTlflMu8E0bDTb5I5gertgwAKmsdUlTQIhZtunzUlD1WSzwcVWPoAVpsrA6vlfCD7oXvRwgg==} + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} engines: {node: ^10 || ^12 || >=14} powershell-utils@0.1.0: @@ -2444,8 +2421,8 @@ packages: peerDependencies: react: ^19.2.8 - react-hook-form@7.83.0: - resolution: {integrity: sha512-AXt8cMCmx5a7u4uvpb2uRFVrWQhllI4pV+LSykxIac/hjt44TnQkmX9BKuQi2i+LDC62esmiLpilkav+kjVf/A==} + react-hook-form@7.84.0: + resolution: {integrity: sha512-+hWvQP6GLco56mDwrbU4XnHix8t1z90ltZsDIrREl+jnQFQxYLX8oAzqe/Xn8nHpmoXTY5M6oEXrAhbP1qevNQ==} engines: {node: '>=18.0.0'} peerDependencies: react: ^16.8.0 || ^17 || ^18 || ^19 @@ -2456,15 +2433,15 @@ packages: react-is@17.0.2: resolution: {integrity: sha512-w2GsyukL62IJnlaff/nRegPQR94C/XXamvMWmSHRJ4y7Ts/4ocGRmTHvOs8PSE6pB3dWOrD/nueuU5sduBsQ4w==} - react-router-dom@7.18.1: - resolution: {integrity: sha512-KaZh+X/6UtEp28x51AUYZDMg9NGoz2ja3dNHa+ta/tk40vCzKhQ/RypCWBMLbmDr6//E24Vv5uPsrqXFozdkAg==} + react-router-dom@7.18.2: + resolution: {integrity: sha512-AIKJ/jgGlFb3EbfCXk5Gzshiwt+l3mqbCrNjmEWMMjqQxNJ3svBa6bgzFyCC2Sw3RA0VWF1kg3uQf2OFhxb8hw==} engines: {node: '>=20.0.0'} peerDependencies: react: '>=18' react-dom: '>=18' - react-router@7.18.1: - resolution: {integrity: sha512-GDLgg3i3uM0aeJO3Fm+TCS+sDQ7gu12T6x0qdTEzcwqEfleci7JwugVNIF3U//0FWKnJT7ptG+20B2jfDqnZAg==} + react-router@7.18.2: + resolution: {integrity: sha512-aUVMjFm3GAPTTZL7oYr5E7ETiqfQCHRLH+B+5afnICvf0r7kkK4eR6SMuwbSTJw/7t+12khT/Kahij49fqOCIg==} engines: {node: '>=20.0.0'} peerDependencies: react: '>=18' @@ -2517,8 +2494,8 @@ packages: resolution: {integrity: sha512-U7WjGVG9sH8tvjW5SmGbQuui75FiyjAX72HX15DwBBwF9dNiQZRQAg9nnPhYy+TUnE0+VcrttuvNI8oSxZcocA==} hasBin: true - rolldown@1.1.5: - resolution: {integrity: sha512-t9z29cJjXf/vxQ8dyhCSpt6H6aSwHTk8cT5I3iy6SMXuFpk5mB6PL6XfC8PCwrPTx93udwKUm9HRteAlTGBLiA==} + rolldown@1.2.3: + resolution: {integrity: sha512-rn9wpmxplLf7NLNyCk9FyWh3FM43DbY8jOzCdEPzH7uflhTftRbCEpqi6Ly2osgoU8OwObtmavMbWLaWy4LX7A==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true @@ -2755,13 +2732,13 @@ packages: peerDependencies: vite: '>8.0.0-0' - vite@8.1.5: - resolution: {integrity: sha512-7ULLwsCdYx/nRyrpiEwvqb5TFHrMVZyBt+rg/OAXT7rgj/z+DtTDyKFeLAdDkubDVDKD8jOsndmy7m55XcfUsw==} + vite@8.2.0: + resolution: {integrity: sha512-pn+CFpM0lwDeKwmOq1ZaBK/9sjorZcgqxki6MbY/jPEVd9vichIlmlD4HmQ5wdP5EgqQCFRaACBxMC7uEGc6lQ==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: '@types/node': ^20.19.0 || >=22.12.0 - '@vitejs/devtools': ^0.3.0 + '@vitejs/devtools': ^0.4.0 esbuild: '>=0.28.1' jiti: '>=1.21.0' less: ^4.0.0 @@ -3163,11 +3140,11 @@ snapshots: '@bcoe/v8-coverage@1.0.2': {} - '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@ark-ui/react': 5.37.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@emotion/is-prop-valid': 1.4.0 - '@emotion/react': 11.14.0(@types/react@19.2.17)(react@19.2.8) + '@emotion/react': 11.14.0(@types/react@19.2.18)(react@19.2.8) '@emotion/serialize': 1.3.3 '@emotion/use-insertion-effect-with-fallbacks': 1.2.0(react@19.2.8) '@emotion/utils': 1.4.2 @@ -3176,22 +3153,6 @@ snapshots: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - '@emnapi/core@1.11.1': - dependencies: - '@emnapi/wasi-threads': 1.2.2 - tslib: 2.8.1 - optional: true - - '@emnapi/runtime@1.11.1': - dependencies: - tslib: 2.8.1 - optional: true - - '@emnapi/wasi-threads@1.2.2': - dependencies: - tslib: 2.8.1 - optional: true - '@emotion/babel-plugin@11.13.5': dependencies: '@babel/helper-module-imports': 7.29.7 @@ -3224,7 +3185,7 @@ snapshots: '@emotion/memoize@0.9.0': {} - '@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8)': + '@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8)': dependencies: '@babel/runtime': 7.29.7 '@emotion/babel-plugin': 11.13.5 @@ -3236,7 +3197,7 @@ snapshots: hoist-non-react-statics: 3.3.2 react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 transitivePeerDependencies: - supports-color @@ -3321,10 +3282,10 @@ snapshots: '@floating-ui/utils@0.2.12': {} - '@hey-api/client-axios@0.9.1(@hey-api/openapi-ts@0.99.0(magicast@0.5.3)(typescript@6.0.3))(axios@1.18.1)': + '@hey-api/client-axios@0.9.1(@hey-api/openapi-ts@0.99.0(magicast@0.5.3)(typescript@6.0.3))(axios@1.19.0)': dependencies: '@hey-api/openapi-ts': 0.99.0(magicast@0.5.3)(typescript@6.0.3) - axios: 1.18.1 + axios: 1.19.0 '@hey-api/codegen-core@0.9.1(magicast@0.5.3)': dependencies: @@ -3423,66 +3384,52 @@ snapshots: '@lukeed/ms@2.0.2': {} - '@napi-rs/wasm-runtime@1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1)': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@tybys/wasm-util': 0.10.3 - optional: true - - '@oxc-project/types@0.139.0': {} + '@oxc-project/types@0.143.0': {} '@pandacss/is-valid-prop@1.11.5': {} '@pkgr/core@0.3.6': {} - '@rolldown/binding-android-arm64@1.1.5': + '@rolldown/binding-android-arm64@1.2.3': optional: true - '@rolldown/binding-darwin-arm64@1.1.5': + '@rolldown/binding-darwin-arm64@1.2.3': optional: true - '@rolldown/binding-darwin-x64@1.1.5': + '@rolldown/binding-darwin-x64@1.2.3': optional: true - '@rolldown/binding-freebsd-x64@1.1.5': + '@rolldown/binding-freebsd-x64@1.2.3': optional: true - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': optional: true - '@rolldown/binding-linux-arm64-gnu@1.1.5': + '@rolldown/binding-linux-arm64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-arm64-musl@1.1.5': + '@rolldown/binding-linux-arm64-musl@1.2.3': optional: true - '@rolldown/binding-linux-ppc64-gnu@1.1.5': + '@rolldown/binding-linux-ppc64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-s390x-gnu@1.1.5': + '@rolldown/binding-linux-s390x-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-gnu@1.1.5': + '@rolldown/binding-linux-x64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-musl@1.1.5': + '@rolldown/binding-linux-x64-musl@1.2.3': optional: true - '@rolldown/binding-openharmony-arm64@1.1.5': + '@rolldown/binding-openharmony-arm64@1.2.3': optional: true - '@rolldown/binding-wasm32-wasi@1.1.5': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@napi-rs/wasm-runtime': 1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1) - optional: true - - '@rolldown/binding-win32-arm64-msvc@1.1.5': + '@rolldown/binding-win32-arm64-msvc@1.2.3': optional: true - '@rolldown/binding-win32-x64-msvc@1.1.5': + '@rolldown/binding-win32-x64-msvc@1.2.3': optional: true '@rolldown/pluginutils@1.0.1': {} @@ -3499,59 +3446,59 @@ snapshots: estraverse: 5.3.0 picomatch: 4.0.4 - '@swc/core-darwin-arm64@1.15.46': + '@swc/core-darwin-arm64@1.15.47': optional: true - '@swc/core-darwin-x64@1.15.46': + '@swc/core-darwin-x64@1.15.47': optional: true - '@swc/core-linux-arm-gnueabihf@1.15.46': + '@swc/core-linux-arm-gnueabihf@1.15.47': optional: true - '@swc/core-linux-arm64-gnu@1.15.46': + '@swc/core-linux-arm64-gnu@1.15.47': optional: true - '@swc/core-linux-arm64-musl@1.15.46': + '@swc/core-linux-arm64-musl@1.15.47': optional: true - '@swc/core-linux-ppc64-gnu@1.15.46': + '@swc/core-linux-ppc64-gnu@1.15.47': optional: true - '@swc/core-linux-s390x-gnu@1.15.46': + '@swc/core-linux-s390x-gnu@1.15.47': optional: true - '@swc/core-linux-x64-gnu@1.15.46': + '@swc/core-linux-x64-gnu@1.15.47': optional: true - '@swc/core-linux-x64-musl@1.15.46': + '@swc/core-linux-x64-musl@1.15.47': optional: true - '@swc/core-win32-arm64-msvc@1.15.46': + '@swc/core-win32-arm64-msvc@1.15.47': optional: true - '@swc/core-win32-ia32-msvc@1.15.46': + '@swc/core-win32-ia32-msvc@1.15.47': optional: true - '@swc/core-win32-x64-msvc@1.15.46': + '@swc/core-win32-x64-msvc@1.15.47': optional: true - '@swc/core@1.15.46(@swc/helpers@0.5.23)': + '@swc/core@1.15.47(@swc/helpers@0.5.23)': dependencies: '@swc/counter': 0.1.3 - '@swc/types': 0.1.27 + '@swc/types': 0.1.28 optionalDependencies: - '@swc/core-darwin-arm64': 1.15.46 - '@swc/core-darwin-x64': 1.15.46 - '@swc/core-linux-arm-gnueabihf': 1.15.46 - '@swc/core-linux-arm64-gnu': 1.15.46 - '@swc/core-linux-arm64-musl': 1.15.46 - '@swc/core-linux-ppc64-gnu': 1.15.46 - '@swc/core-linux-s390x-gnu': 1.15.46 - '@swc/core-linux-x64-gnu': 1.15.46 - '@swc/core-linux-x64-musl': 1.15.46 - '@swc/core-win32-arm64-msvc': 1.15.46 - '@swc/core-win32-ia32-msvc': 1.15.46 - '@swc/core-win32-x64-msvc': 1.15.46 + '@swc/core-darwin-arm64': 1.15.47 + '@swc/core-darwin-x64': 1.15.47 + '@swc/core-linux-arm-gnueabihf': 1.15.47 + '@swc/core-linux-arm64-gnu': 1.15.47 + '@swc/core-linux-arm64-musl': 1.15.47 + '@swc/core-linux-ppc64-gnu': 1.15.47 + '@swc/core-linux-s390x-gnu': 1.15.47 + '@swc/core-linux-x64-gnu': 1.15.47 + '@swc/core-linux-x64-musl': 1.15.47 + '@swc/core-win32-arm64-msvc': 1.15.47 + '@swc/core-win32-ia32-msvc': 1.15.47 + '@swc/core-win32-x64-msvc': 1.15.47 '@swc/helpers': 0.5.23 '@swc/counter@0.1.3': {} @@ -3560,7 +3507,7 @@ snapshots: dependencies: tslib: 2.8.1 - '@swc/types@0.1.27': + '@swc/types@0.1.28': dependencies: '@swc/counter': 0.1.3 @@ -3592,15 +3539,15 @@ snapshots: picocolors: 1.1.1 redent: 3.0.0 - '@testing-library/react@16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@testing-library/react@16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@babel/runtime': 7.28.6 '@testing-library/dom': 10.4.0 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) optionalDependencies: - '@types/react': 19.2.17 - '@types/react-dom': 19.2.3(@types/react@19.2.17) + '@types/react': 19.2.18 + '@types/react-dom': 19.2.4(@types/react@19.2.18) '@trivago/prettier-plugin-sort-imports@6.0.2(prettier@3.9.6)': dependencies: @@ -3622,11 +3569,6 @@ snapshots: path-browserify: 1.0.1 tinyglobby: 0.2.17 - '@tybys/wasm-util@0.10.3': - dependencies: - tslib: 2.8.1 - optional: true - '@types/aria-query@5.0.4': {} '@types/chai@5.2.3': @@ -3640,9 +3582,9 @@ snapshots: '@types/estree@1.0.9': {} - '@types/hoist-non-react-statics@3.3.7(@types/react@19.2.17)': + '@types/hoist-non-react-statics@3.3.7(@types/react@19.2.18)': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 hoist-non-react-statics: 3.3.2 '@types/json-schema@7.0.15': {} @@ -3653,11 +3595,11 @@ snapshots: '@types/parse-json@4.0.2': {} - '@types/react-dom@19.2.3(@types/react@19.2.17)': + '@types/react-dom@19.2.4(@types/react@19.2.18)': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 - '@types/react@19.2.17': + '@types/react@19.2.18': dependencies: csstype: 3.2.3 @@ -3760,11 +3702,11 @@ snapshots: '@typescript-eslint/types': 8.65.0 eslint-visitor-keys: 5.0.1 - '@vitejs/plugin-react-swc@4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': + '@vitejs/plugin-react-swc@4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@rolldown/pluginutils': 1.0.1 - '@swc/core': 1.15.46(@swc/helpers@0.5.23) - vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) + '@swc/core': 1.15.47(@swc/helpers@0.5.23) + vite: 8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) transitivePeerDependencies: - '@swc/helpers' @@ -3780,7 +3722,7 @@ snapshots: obug: 2.1.3 std-env: 4.2.0 tinyrainbow: 3.1.0 - vitest: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) + vitest: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/expect@4.1.10': dependencies: @@ -3791,13 +3733,13 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.0 - '@vitest/mocker@4.1.10(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': + '@vitest/mocker@4.1.10(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) '@vitest/pretty-format@4.1.10': dependencies: @@ -4508,7 +4450,7 @@ snapshots: axe-core@4.10.3: {} - axios@1.18.1: + axios@1.19.0: dependencies: follow-redirects: 1.16.0 form-data: 4.0.6 @@ -5548,7 +5490,7 @@ snapshots: ms@2.1.3: {} - nanoid@3.3.16: {} + nanoid@3.3.18: {} natural-compare@1.4.0: {} @@ -5688,9 +5630,9 @@ snapshots: possible-typed-array-names@1.1.0: {} - postcss@8.5.23: + postcss@8.5.26: dependencies: - nanoid: 3.3.16 + nanoid: 3.3.18 picocolors: 1.1.1 source-map-js: 1.2.1 @@ -5733,9 +5675,9 @@ snapshots: defu: 6.1.7 destr: 2.0.5 - react-cookie@8.1.2(@types/react@19.2.17)(react@19.2.8): + react-cookie@8.1.2(@types/react@19.2.18)(react@19.2.8): dependencies: - '@types/hoist-non-react-statics': 3.3.7(@types/react@19.2.17) + '@types/hoist-non-react-statics': 3.3.7(@types/react@19.2.18) hoist-non-react-statics: 3.3.2 react: 19.2.8 universal-cookie: 8.1.2 @@ -5747,7 +5689,7 @@ snapshots: react: 19.2.8 scheduler: 0.27.0 - react-hook-form@7.83.0(react@19.2.8): + react-hook-form@7.84.0(react@19.2.8): dependencies: react: 19.2.8 @@ -5755,13 +5697,13 @@ snapshots: react-is@17.0.2: {} - react-router-dom@7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + react-router-dom@7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - react-router: 7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + react-router: 7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) - react-router@7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + react-router@7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: cookie: 1.1.1 react: 19.2.8 @@ -5821,26 +5763,25 @@ snapshots: path-parse: 1.0.7 supports-preserve-symlinks-flag: 1.0.0 - rolldown@1.1.5: + rolldown@1.2.3: dependencies: - '@oxc-project/types': 0.139.0 + '@oxc-project/types': 0.143.0 '@rolldown/pluginutils': 1.0.1 optionalDependencies: - '@rolldown/binding-android-arm64': 1.1.5 - '@rolldown/binding-darwin-arm64': 1.1.5 - '@rolldown/binding-darwin-x64': 1.1.5 - '@rolldown/binding-freebsd-x64': 1.1.5 - '@rolldown/binding-linux-arm-gnueabihf': 1.1.5 - '@rolldown/binding-linux-arm64-gnu': 1.1.5 - '@rolldown/binding-linux-arm64-musl': 1.1.5 - '@rolldown/binding-linux-ppc64-gnu': 1.1.5 - '@rolldown/binding-linux-s390x-gnu': 1.1.5 - '@rolldown/binding-linux-x64-gnu': 1.1.5 - '@rolldown/binding-linux-x64-musl': 1.1.5 - '@rolldown/binding-openharmony-arm64': 1.1.5 - '@rolldown/binding-wasm32-wasi': 1.1.5 - '@rolldown/binding-win32-arm64-msvc': 1.1.5 - '@rolldown/binding-win32-x64-msvc': 1.1.5 + '@rolldown/binding-android-arm64': 1.2.3 + '@rolldown/binding-darwin-arm64': 1.2.3 + '@rolldown/binding-darwin-x64': 1.2.3 + '@rolldown/binding-freebsd-x64': 1.2.3 + '@rolldown/binding-linux-arm-gnueabihf': 1.2.3 + '@rolldown/binding-linux-arm64-gnu': 1.2.3 + '@rolldown/binding-linux-arm64-musl': 1.2.3 + '@rolldown/binding-linux-ppc64-gnu': 1.2.3 + '@rolldown/binding-linux-s390x-gnu': 1.2.3 + '@rolldown/binding-linux-x64-gnu': 1.2.3 + '@rolldown/binding-linux-x64-musl': 1.2.3 + '@rolldown/binding-openharmony-arm64': 1.2.3 + '@rolldown/binding-win32-arm64-msvc': 1.2.3 + '@rolldown/binding-win32-x64-msvc': 1.2.3 run-applescript@7.1.0: {} @@ -6116,16 +6057,16 @@ snapshots: dependencies: punycode: 2.3.1 - vite-plugin-css-injected-by-js@5.0.2(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): + vite-plugin-css-injected-by-js@5.0.2(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: - vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) - vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0): + vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0): dependencies: lightningcss: 1.33.0 picomatch: 4.0.5 - postcss: 8.5.23 - rolldown: 1.1.5 + postcss: 8.5.26 + rolldown: 1.2.3 tinyglobby: 0.2.17 optionalDependencies: '@types/node': 26.1.2 @@ -6133,10 +6074,10 @@ snapshots: jiti: 2.7.0 yaml: 2.9.0 - vitest@4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): + vitest@4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.10 - '@vitest/mocker': 4.1.10(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) + '@vitest/mocker': 4.1.10(vite@8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/pretty-format': 4.1.10 '@vitest/runner': 4.1.10 '@vitest/snapshot': 4.1.10 @@ -6153,7 +6094,7 @@ snapshots: tinyexec: 1.2.4 tinyglobby: 0.2.17 tinyrainbow: 3.1.0 - vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) why-is-node-running: 2.3.0 optionalDependencies: '@types/node': 26.1.2 diff --git a/airflow-core/src/airflow/api_fastapi/common/parameters.py b/airflow-core/src/airflow/api_fastapi/common/parameters.py index 76fe643fd6b47..dfc255f468301 100644 --- a/airflow-core/src/airflow/api_fastapi/common/parameters.py +++ b/airflow-core/src/airflow/api_fastapi/common/parameters.py @@ -39,6 +39,7 @@ from sqlalchemy import Column, String, and_, func, not_, or_, select as sql_select, true as sql_true from sqlalchemy.ext.compiler import compiles from sqlalchemy.inspection import inspect +from sqlalchemy.orm import aliased from sqlalchemy.sql.functions import FunctionElement from airflow._shared.timezones import timezone @@ -1480,10 +1481,13 @@ def to_orm(self, select: Select) -> Select: if self.value is None and self.skip_none: return select - # EXISTS resolves each Dag via the (dag_id, state) index instead of scanning every run in the state. + # Alias DagRun so this EXISTS subquery cannot auto-correlate to a DagRun the outer query + # may already reference (e.g. the last_dag_run_state filter), which would strip the + # subquery's FROM and raise. EXISTS resolves each Dag via the (dag_id, state) index. + any_run = aliased(DagRun) has_run_in_state = ( - sql_select(DagRun.dag_id) - .where(DagRun.dag_id == DagModel.dag_id, DagRun.state == self.value) + sql_select(any_run.dag_id) + .where(any_run.dag_id == DagModel.dag_id, any_run.state == self.value) .exists() ) return select.where(has_run_in_state) diff --git a/airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml b/airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml index 32f0f6f173511..33c989d0ca053 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml +++ b/airflow-core/src/airflow/api_fastapi/core_api/openapi/_private_ui.yaml @@ -1330,6 +1330,190 @@ paths: minimum: 0 default: 0 title: Offset + - name: from_date_gte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: From Date Gte + - name: from_date_gt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: From Date Gt + - name: from_date_lte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: From Date Lte + - name: from_date_lt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: From Date Lt + - name: to_date_gte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: To Date Gte + - name: to_date_gt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: To Date Gt + - name: to_date_lte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: To Date Lte + - name: to_date_lt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: To Date Lt + - name: created_at_gte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Created At Gte + - name: created_at_gt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Created At Gt + - name: created_at_lte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Created At Lte + - name: created_at_lt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Created At Lt + - name: completed_at_gte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Completed At Gte + - name: completed_at_gt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Completed At Gt + - name: completed_at_lte + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Completed At Lte + - name: completed_at_lt + in: query + required: false + schema: + anyOf: + - type: string + format: date-time + - type: 'null' + title: Completed At Lt + - name: max_active_runs_gte + in: query + required: false + schema: + anyOf: + - type: number + - type: 'null' + title: Max Active Runs Gte + - name: max_active_runs_gt + in: query + required: false + schema: + anyOf: + - type: number + - type: 'null' + title: Max Active Runs Gt + - name: max_active_runs_lte + in: query + required: false + schema: + anyOf: + - type: number + - type: 'null' + title: Max Active Runs Lte + - name: max_active_runs_lt + in: query + required: false + schema: + anyOf: + - type: number + - type: 'null' + title: Max Active Runs Lt + - name: reprocess_behavior + in: query + required: false + schema: + anyOf: + - type: string + - type: 'null' + title: Reprocess Behavior - name: order_by in: query required: false diff --git a/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml b/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml index fea2773828ced..92edb82d7f915 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml +++ b/airflow-core/src/airflow/api_fastapi/core_api/openapi/v2-rest-api-generated.yaml @@ -7027,6 +7027,12 @@ paths: schema: $ref: '#/components/schemas/HTTPExceptionResponse' description: Not Found + '409': + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPExceptionResponse' + description: Conflict '422': description: Validation Error content: diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py index 0a6818648cd68..830d4edaeedac 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/assets.py @@ -86,6 +86,7 @@ AssetWatcherModel, TaskOutletAssetReference, ) +from airflow.models.dag import DagModel from airflow.models.dag_version import DagVersion from airflow.typing_compat import Unpack from airflow.utils.state import DagRunState @@ -459,7 +460,10 @@ def materialize_asset( if not get_auth_manager().is_authorized_dag( method="POST", access_entity=DagAccessEntity.RUN, - details=DagDetails(id=dag_id), + # The Dag is resolved from the asset here rather than named by the caller, so its team has + # to be looked up too. A team-aware auth manager distinguishes a team-scoped Dag from a + # global one by this field, so leaving it None asks the wrong question. + details=DagDetails(id=dag_id, team_name=DagModel.get_team_name(dag_id, session=session)), user=user, ): raise HTTPException( diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/config.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/config.py index 9f4b3359508a1..ea6af1e7e20a7 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/config.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/config.py @@ -153,9 +153,7 @@ def get_config_value( ) section_l, option_l = section.lower(), option.lower() - if (section_l, option_l) in conf.sensitive_config_values or _is_per_key_sensitive_option( - section_l, option_l - ): + if conf.is_sensitive_option(section_l, option_l) or _is_per_key_sensitive_option(section_l, option_l): value = "< hidden >" else: value = conf.get(section, option) diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py index 1b857df06ce57..e5d06587cb550 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py @@ -869,7 +869,10 @@ def wait_dag_run_until_finished( if not get_auth_manager().is_authorized_dag( method="GET", access_entity=DagAccessEntity.XCOM, - details=DagDetails(id=dag_id), + # The route dependency above already authorizes RUN access with the Dag's team resolved; + # this second, XCom-specific check has to resolve it the same way, or the two checks ask + # a team-aware auth manager about differently-scoped resources. + details=DagDetails(id=dag_id, team_name=DagModel.get_team_name(dag_id, session=session)), user=user, ): if result_task_ids: diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/xcom.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/xcom.py index 13c7f78ed28f5..40bb97cfc9fee 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/public/xcom.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/public/xcom.py @@ -237,6 +237,7 @@ def get_xcom_entries( [ status.HTTP_400_BAD_REQUEST, status.HTTP_404_NOT_FOUND, + status.HTTP_409_CONFLICT, ] ), dependencies=[ @@ -269,10 +270,9 @@ def create_xcom_entry( # Validate Dag Run ID if not dag_run: - if not dag_run: - raise HTTPException( - status.HTTP_404_NOT_FOUND, f"Dag Run with ID: `{dag_run_id}` not found for dag: `{dag_id}`" - ) + raise HTTPException( + status.HTTP_404_NOT_FOUND, f"Dag Run with ID: `{dag_run_id}` not found for dag: `{dag_id}`" + ) # Check existing XCom already_existing_query = XComModel.get_many( diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/backfills.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/backfills.py index 02583a8355b9d..f3a82f4e3aab6 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/backfills.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/backfills.py @@ -28,8 +28,11 @@ FilterParam, QueryLimit, QueryOffset, + RangeFilter, SortParam, + datetime_range_filter_factory, filter_param_factory, + float_range_filter_factory, ) from airflow.api_fastapi.common.router import AirflowRouter from airflow.api_fastapi.core_api.datamodels.backfills import BackfillCollectionResponse, BackfillResponse @@ -52,6 +55,14 @@ def list_backfills_ui( limit: QueryLimit, offset: QueryOffset, + start_date_range: Annotated[RangeFilter, Depends(datetime_range_filter_factory("from_date", Backfill))], + end_date_range: Annotated[RangeFilter, Depends(datetime_range_filter_factory("to_date", Backfill))], + created_at: Annotated[RangeFilter, Depends(datetime_range_filter_factory("created_at", Backfill))], + completed_at: Annotated[RangeFilter, Depends(datetime_range_filter_factory("completed_at", Backfill))], + max_active_runs: Annotated[RangeFilter, Depends(float_range_filter_factory("max_active_runs", Backfill))], + reprocess_behavior: Annotated[ + FilterParam, Depends(filter_param_factory(Backfill.reprocess_behavior, str | None)) + ], order_by: Annotated[ SortParam, Depends(SortParam(["id"], Backfill).dynamic_depends()), @@ -66,7 +77,17 @@ def list_backfills_ui( ) -> BackfillCollectionResponse: select_stmt, total_entries = paginated_select( statement=select(Backfill).options(joinedload(Backfill.dag_model)), - filters=[dag_id, active, readable_backfills_filter], + filters=[ + dag_id, + active, + readable_backfills_filter, + start_date_range, + end_date_range, + created_at, + completed_at, + max_active_runs, + reprocess_behavior, + ], order_by=order_by, offset=offset, limit=limit, diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py index 13326f68a33db..48bd5971461c1 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py @@ -84,14 +84,7 @@ def _get_latest_serdag(dag_id, session): - serdag = session.scalar( - select(SerializedDagModel) - .where( - SerializedDagModel.dag_id == dag_id, - ) - .order_by(SerializedDagModel.id.desc()) - .limit(1) - ) + serdag = session.scalar(SerializedDagModel.latest_item_select_object(dag_id)) if not serdag: raise HTTPException( status.HTTP_404_NOT_FOUND, diff --git a/airflow-core/src/airflow/api_fastapi/core_api/security.py b/airflow-core/src/airflow/api_fastapi/core_api/security.py index 96478eb5a1483..e223dd207360c 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/security.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/security.py @@ -27,6 +27,7 @@ from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer, OAuth2PasswordBearer from itsdangerous import BadSignature, URLSafeSerializer from jwt import ExpiredSignatureError, InvalidTokenError +from pydantic import NonNegativeInt, TypeAdapter, ValidationError from sqlalchemy import or_, select from sqlalchemy.orm import Session @@ -390,6 +391,12 @@ def depends_readable_event_logs_filter( ] +# The type the backfill routes declare for the `backfill_id` path parameter. Shared with +# `requires_access_backfill` so the authorization decision parses the id exactly as the handler +# does; see the comment there for why any divergence is a cross-Dag authorization bypass. +_BACKFILL_ID_ADAPTER: TypeAdapter[NonNegativeInt] = TypeAdapter(NonNegativeInt) + + def requires_access_backfill( method: ResourceMethod, ) -> Callable[[Request, BaseUser, Session], Coroutine[Any, Any, None]]: @@ -405,8 +412,14 @@ async def inner( # Try to retrieve the dag_id from the backfill_id path param backfill_id_raw = request.path_params.get("backfill_id") try: - backfill_id = int(backfill_id_raw) if backfill_id_raw is not None else None - except ValueError: + # Must parse exactly as the handler does (e.g. pydantic's lax mode coerces "1.0" to 1 + # where int() raises), or the two can authorize and act on different backfills. + backfill_id = ( + _BACKFILL_ID_ADAPTER.validate_python(backfill_id_raw) if backfill_id_raw is not None else None + ) + except ValidationError: + # Rejected by the endpoint's parser too, so the handler cannot run: FastAPI answers + # 422 before it is reached. Left as None, preserving that response. backfill_id = None if backfill_id is not None: @@ -414,6 +427,10 @@ async def inner( dag_id = backfill.dag_id if backfill else None # Try to retrieve the dag_id from the request body (POST backfill) + # TODO: a backfill_id that parses but matches no row also lands here, so an unknown + # backfill is authorized against the body's dag_id and answers 404 where an unauthorized + # one answers 403 - disclosing which ids exist. Not exploitable for a cross-Dag action; + # tracked at https://github.com/apache/airflow/issues/71080 if dag_id is None: # Not a json body, ignore with suppress(JSONDecodeError): diff --git a/airflow-core/src/airflow/api_fastapi/core_api/services/public/config.py b/airflow-core/src/airflow/api_fastapi/core_api/services/public/config.py index d805827d8d61d..e212a838c4673 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/services/public/config.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/services/public/config.py @@ -31,6 +31,11 @@ # (e.g. Vault role_id / secret_id) as the registered ``backend_kwargs`` # option, so they need the same redaction treatment when # ``display_sensitive=False``. +# These match literal section names only, so a team scoped spelling of the same option -- the +# ``[=secrets]`` section, or ``AIRFLOW_____SECRETS__BACKEND_KWARG__*``, which is +# reported under a section named after the team -- is not recognised. Nothing leaks today because +# the secrets backend is not team aware; tracked at +# https://github.com/apache/airflow/issues/71037 _PER_KEY_SENSITIVE_PREFIXES: dict[str, str] = { "secrets": "backend_kwarg__", "workers": "secrets_backend_kwarg__", diff --git a/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py b/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py index 34c3dc35406f8..41ecf49b053fb 100644 --- a/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py +++ b/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py @@ -21,7 +21,7 @@ import itertools import json from collections import defaultdict -from collections.abc import Iterator +from collections.abc import Callable, Iterator, Sequence from typing import TYPE_CHECKING, Annotated, Any, NoReturn, cast from uuid import UUID @@ -450,8 +450,9 @@ def ti_update_state( data["_rendered_map_index"] = data.pop("rendered_map_index") query = update(TI).where(TI.id == task_instance_id).values(data) + asset_callbacks: Sequence[Callable[[], None]] = () try: - query, updated_state = _create_ti_state_update_query_and_update_state( + query, updated_state, asset_callbacks = _create_ti_state_update_query_and_update_state( ti_patch_payload=ti_patch_payload, task_instance_id=task_instance_id, session=session, @@ -468,6 +469,7 @@ def ti_update_state( "Error updating Task Instance state. Setting the task to failed.", payload=ti_patch_payload, ) + session.rollback() ti = session.get(TI, task_instance_id, with_for_update={"of": TI}) if session.bind is not None: query = TI.duration_expression_update(timezone.utcnow(), query, session.bind) @@ -529,6 +531,12 @@ def ti_update_state( task_id=task_id, ) + # Release the task_instance row lock before running listener callbacks. + session.commit() + + for callback in asset_callbacks: + callback() + def _emit_task_span(ti, state): # just to be safe @@ -626,7 +634,8 @@ def _create_ti_state_update_query_and_update_state( session: SessionDep, dag_bag: DagBagDep, dag_id: str, -) -> tuple[Update, TaskInstanceState]: +) -> tuple[Update, TaskInstanceState, Sequence[Callable[[], None]]]: + asset_callbacks: Sequence[Callable[[], None]] = () if isinstance(ti_patch_payload, (TITerminalStatePayload, TIRetryStatePayload, TISuccessStatePayload)): ti = session.get(TI, task_instance_id, with_for_update={"of": TI}) updated_state = TaskInstanceState(ti_patch_payload.state.value) @@ -657,7 +666,7 @@ def _create_ti_state_update_query_and_update_state( query = query.values(retry_delay_override=retry_delay_override, retry_reason=retry_reason) elif isinstance(ti_patch_payload, TISuccessStatePayload): if ti is not None: - TI.register_asset_changes_in_db( + asset_callbacks = TI.register_asset_changes_in_db( ti, ti_patch_payload.task_outlets, ti_patch_payload.outlet_events, @@ -768,7 +777,7 @@ def _create_ti_state_update_query_and_update_state( ti = session.get(TI, task_instance_id, with_for_update={"of": TI}) if ti is not None: _handle_fail_fast_for_dag(ti=ti, dag_id=dag_id, session=session, dag_bag=dag_bag) - return query, TaskInstanceState.FAILED + return query, TaskInstanceState.FAILED, () actual_start_date = timezone.utcnow() session.add( @@ -790,7 +799,7 @@ def _create_ti_state_update_query_and_update_state( else: raise ValueError(f"Unexpected Payload Type {type(ti_patch_payload)}") - return query, updated_state + return query, updated_state, asset_callbacks @ti_id_router.patch( diff --git a/airflow-core/src/airflow/api_fastapi/logging/decorators.py b/airflow-core/src/airflow/api_fastapi/logging/decorators.py index 7c32c70a5ca3d..22acc54b13076 100644 --- a/airflow-core/src/airflow/api_fastapi/logging/decorators.py +++ b/airflow-core/src/airflow/api_fastapi/logging/decorators.py @@ -45,8 +45,51 @@ def _sanitize_for_stdlib_log(value: str) -> str: return value.replace("\r", " ").replace("\n", " ") +def _mask_bulk_entities(extra_fields, mask_entity): + """ + Apply per-entity masking to a bulk request body. + + A ``BulkBody`` has exactly one top-level field, ``actions``; the entities carrying the + secrets sit two levels down, in ``actions[].entities[]``. The per-entity maskers below + inspect top-level key names, so handing them a bulk body means they see only the key + ``actions`` and pass its whole payload through untouched. Reach the entities first. + + Returns ``None`` when the body is not bulk-shaped, so callers fall back to flat masking. + """ + actions = extra_fields.get("actions") + if not isinstance(actions, list): + return None + + masked_actions = [] + for action in actions: + if not isinstance(action, dict): + masked_actions.append(action) + continue + entities = action.get("entities") + if not isinstance(entities, list): + masked_actions.append(action) + continue + # ``delete`` actions may list bare id/key strings rather than entity objects; + # those carry no secret and are left as they are. + masked_actions.append( + { + **action, + "entities": [mask_entity(e) if isinstance(e, dict) else e for e in entities], + } + ) + return {**extra_fields, "actions": masked_actions} + + def _mask_connection_fields(extra_fields): - """Mask connection fields.""" + """Mask connection fields, for either a single-entity or a bulk request body.""" + bulk = _mask_bulk_entities(extra_fields, _mask_connection_entity) + if bulk is not None: + return bulk + return _mask_connection_entity(extra_fields) + + +def _mask_connection_entity(extra_fields): + """Mask the fields of one connection.""" result = {} for k, v in extra_fields.items(): if k == "extra" and v: @@ -59,7 +102,11 @@ def _mask_connection_fields(extra_fields): result[k] = {ek: "***" for ek in parsed_extra} else: result[k] = "Expected JSON object in `extra` field, got non-dict JSON" - except json.JSONDecodeError: + except (json.JSONDecodeError, TypeError): + # ``extra`` is declared as a string, but this runs on the raw body before + # validation, so it can arrive as any JSON type -- a number or an already-decoded + # object makes ``json.loads`` raise TypeError rather than JSONDecodeError. Both + # are recorded without the value, instead of raising out of the audit-log path. result[k] = "Encountered non-JSON in `extra` field" else: result[k] = secrets_masker.redact(v, k) @@ -67,6 +114,14 @@ def _mask_connection_fields(extra_fields): def _mask_variable_fields(extra_fields): + """Mask variable values, for either a single-entity or a bulk request body.""" + bulk = _mask_bulk_entities(extra_fields, _mask_variable_entity) + if bulk is not None: + return bulk + return _mask_variable_entity(extra_fields) + + +def _mask_variable_entity(extra_fields): """ Mask the variable value. diff --git a/airflow-core/src/airflow/assets/manager.py b/airflow-core/src/airflow/assets/manager.py index d0f8470b17c94..a417ad96eae23 100644 --- a/airflow-core/src/airflow/assets/manager.py +++ b/airflow-core/src/airflow/assets/manager.py @@ -17,12 +17,13 @@ # under the License. from __future__ import annotations -from collections.abc import Collection, Iterable +from collections.abc import Callable, Collection, Iterable from contextlib import contextmanager +from functools import partial from typing import TYPE_CHECKING import structlog -from sqlalchemy import exc, or_, select +from sqlalchemy import exc, insert, or_, select from sqlalchemy.orm import joinedload from airflow._shared.observability.metrics import stats @@ -41,12 +42,12 @@ DagScheduleAssetUriReference, PartitionedAssetKeyLog, TaskOutletAssetReference, + asset_alias_asset_event_association_table, ) from airflow.models.log import Log from airflow.timetables.base import compute_rollup_fingerprint from airflow.utils.helpers import is_container, prune_dict from airflow.utils.log.logging_mixin import LoggingMixin -from airflow.utils.session import create_session from airflow.utils.sqlalchemy import get_dialect_name, with_row_locks if TYPE_CHECKING: @@ -120,38 +121,6 @@ def _lock_asset_model( yield -def _create_asset_event(*, session: Session, **event_kwargs) -> AssetEvent: - """ - Persist an :class:`AssetEvent` row and return it, bound to *session*. - - On SQLite the event is added directly to the caller's *session* and - flushed. SQLite serialises writes at the database-file level: opening - a second connection here would compete with any write locks the - caller's transaction already holds (for example, an UPDATE on - ``dag_run`` flushed earlier in ``register_asset_changes_in_db``) and - deadlock with ``database is locked``. - - On Postgres/MySQL a short-lived independent session is used so the - row is committed — and therefore visible to the scheduler's session - via MVCC — before the caller continues. The committed row is then - re-loaded into the caller's *session* so subsequent relationship - operations work correctly. - """ - if get_dialect_name(session) == "sqlite": - asset_event = AssetEvent(**event_kwargs) - session.add(asset_event) - session.flush() - return asset_event - - with create_session(scoped=False) as ae_session: - asset_event = AssetEvent(**event_kwargs) - ae_session.add(asset_event) - ae_session.flush() - asset_event_id = asset_event.id - - return session.get_one(AssetEvent, asset_event_id) - - class AssetManager(LoggingMixin): """ A pluggable class that manages operations for assets. @@ -314,6 +283,7 @@ def register_asset_change( api_user_teams: set[str] | None = None, api_allow_consumer_teams: list[str] | None = None, api_allow_global_consumers: bool = True, + callback_sink: list[Callable[[], None]] | None = None, **kwargs, ) -> AssetEvent | None: """ @@ -339,6 +309,8 @@ def register_asset_change( Only used when source_is_api=True. :param api_allow_global_consumers: Whether teamless consumers are allowed for an API-triggered event. Only used when source_is_api=True. Defaults to True. + :param callback_sink: If specified, registration callbacks are added + into the list instead of executed inline. """ from airflow.models.dag import DagModel @@ -374,23 +346,35 @@ def register_asset_change( source_run_id=task_instance.run_id, source_map_index=task_instance.map_index, ) - asset_event = _create_asset_event(session=session, **event_kwargs) + + asset_event = AssetEvent(**event_kwargs) + session.add(asset_event) + session.flush() dags_to_queue_from_asset = {ref.dag for ref in asset_model.scheduled_dags if not ref.dag.is_paused} dags_to_queue_from_asset_alias = set() if source_alias_names: - asset_alias_models: Iterable[AssetAliasModel] = session.scalars( - select(AssetAliasModel) - .where(AssetAliasModel.name.in_(source_alias_names)) - .options( - joinedload(AssetAliasModel.scheduled_dags).joinedload(DagScheduleAssetAliasReference.dag) + asset_alias_models = ( + session.scalars( + select(AssetAliasModel) + .where(AssetAliasModel.name.in_(source_alias_names)) + .options( + joinedload(AssetAliasModel.scheduled_dags).joinedload( + DagScheduleAssetAliasReference.dag + ) + ) ) - ).unique() + .unique() + .all() + ) for asset_alias_model in asset_alias_models: - asset_alias_model.asset_events.append(asset_event) - session.add(asset_alias_model) + session.execute( + insert(asset_alias_asset_event_association_table).values( + alias_id=asset_alias_model.id, event_id=asset_event.id + ) + ) dags_to_queue_from_asset_alias |= { alias_ref.dag @@ -416,20 +400,23 @@ def register_asset_change( ) asset = asset_model.to_serialized() - cls.notify_asset_changed(asset=asset) - cls.nofity_asset_event_emitted( - asset_event=ListenerAssetEvent( - asset=asset, - extra=asset_event.extra, - source_dag_id=asset_event.source_dag_id, - source_task_id=asset_event.source_task_id, - source_run_id=asset_event.source_run_id, - source_map_index=asset_event.source_map_index, - source_aliases=[aam.to_serialized() for aam in asset_alias_models], - partition_key=partition_key, - partition_date=partition_date, - ) + listener_asset_event = ListenerAssetEvent( + asset=asset, + extra=asset_event.extra, + source_dag_id=asset_event.source_dag_id, + source_task_id=asset_event.source_task_id, + source_run_id=asset_event.source_run_id, + source_map_index=asset_event.source_map_index, + source_aliases=[aam.to_serialized() for aam in asset_alias_models], + partition_key=partition_key, + partition_date=partition_date, ) + if callback_sink is None: + cls.notify_asset_changed(asset=asset) + cls.nofity_asset_event_emitted(asset_event=listener_asset_event) + else: + callback_sink.append(partial(cls.notify_asset_changed, asset=asset)) + callback_sink.append(partial(cls.nofity_asset_event_emitted, asset_event=listener_asset_event)) team_name = None if task_instance and conf.getboolean("core", "multi_team"): @@ -544,25 +531,7 @@ def _queue_dagruns( if not non_partitioned_dags or partition_key is not None: return None - # Possible race condition: if multiple dags or multiple (usually - # mapped) tasks update the same asset, this can fail with a unique - # constraint violation. - # - # Where the dialect supports a single-statement "insert, update on - # conflict" we use it; it is atomic, avoids the per-row SAVEPOINT churn, - # and holds locks for far less time (which on MySQL/InnoDB also makes the - # concurrent fan-out much less deadlock-prone). Otherwise we "fallback" to - # a nested transaction per row. Either way the rows are added in the same - # transaction where `ti.state` is changed. - dialect_name = get_dialect_name(session) - if TYPE_CHECKING: - assert dialect_name is not None - if dialect_name == "mysql": - return cls._queue_dagruns_nonpartitioned_mysql(asset_id, non_partitioned_dags, event, session) - # PostgreSQL and SQLite both support ON CONFLICT DO UPDATE. - return cls._queue_dagruns_nonpartitioned_conflict_update( - asset_id, non_partitioned_dags, event, session, dialect_name - ) + return cls._queue_dagruns_nonpartitioned(asset_id, non_partitioned_dags, event, session) @classmethod def _queue_partitioned_dags( @@ -827,71 +796,38 @@ def _get_or_create_apdr( return apdr @classmethod - def _queue_dagruns_nonpartitioned_slow_path( + def _queue_dagruns_nonpartitioned( cls, asset_id: int, dags_to_queue: set[DagModel], event: AssetEvent, session: Session ) -> None: - def _queue_dagrun_if_needed(dag: DagModel) -> str | None: - item = AssetDagRunQueue(target_dag_id=dag.dag_id, asset_id=asset_id, created_at=event.timestamp) - # Don't error whole transaction when a single RunQueue item conflicts. - # https://docs.sqlalchemy.org/en/20/orm/session_transaction.html#using-savepoint - try: - with session.begin_nested(): - existing = session.get( - AssetDagRunQueue, {"target_dag_id": dag.dag_id, "asset_id": asset_id} - ) - if existing and existing.created_at >= event.timestamp: - cls.logger().debug("Skipping record %s due to newer timestamp", item) - return dag.dag_id # already queued with a newer timestamp - session.merge(item) - except exc.IntegrityError: - cls.logger().debug("Skipping record %s", item, exc_info=True) - return dag.dag_id - - queued_results = (_queue_dagrun_if_needed(dag) for dag in dags_to_queue) - if queued_dag_ids := [r for r in queued_results if r is not None]: - cls.logger().debug("consuming dag ids %s", queued_dag_ids) - - @classmethod - def _queue_dagruns_nonpartitioned_mysql( - cls, asset_id: int, dags_to_queue: set[DagModel], event: AssetEvent, session: Session - ) -> None: - from sqlalchemy import case - from sqlalchemy.dialects.mysql import insert - - values = [{"target_dag_id": dag.dag_id} for dag in dags_to_queue] - stmt = insert(AssetDagRunQueue).values(asset_id=asset_id, created_at=event.timestamp) - - update_stmt = stmt.on_duplicate_key_update( - created_at=case( - (stmt.inserted.created_at >= AssetDagRunQueue.created_at, stmt.inserted.created_at), - else_=AssetDagRunQueue.created_at, + if not dags_to_queue: + return + values = [ + {"asset_id": asset_id, "target_dag_id": dag.dag_id, "asset_event_id": event.id} + for dag in dags_to_queue + ] + + if (dialect_name := get_dialect_name(session)) == "mysql": + from sqlalchemy.dialects.mysql import insert as my_insert + + session.execute( + # The on-dup update a no-op since we are always working on ADRQs + # for the asset. It may not hold if you change e.g. ADRQ schema. + my_insert(AssetDagRunQueue).on_duplicate_key_update(asset_id=AssetDagRunQueue.asset_id), + values, ) - ) - session.execute(update_stmt, values) + return - @classmethod - def _queue_dagruns_nonpartitioned_conflict_update( - cls, - asset_id: int, - dags_to_queue: set[DagModel], - event: AssetEvent, - session: Session, - dialect_name: str, - ) -> None: - """Handle ON CONFLICT DO UPDATE upsert for dialects that support it (postgresql, sqlite).""" if dialect_name == "postgresql": from sqlalchemy.dialects.postgresql import insert else: from sqlalchemy.dialects.sqlite import insert # type: ignore[assignment] - values = [{"target_dag_id": dag.dag_id} for dag in dags_to_queue] - stmt = insert(AssetDagRunQueue).values(asset_id=asset_id, created_at=event.timestamp) - update_stmt = stmt.on_conflict_do_update( - index_elements=["asset_id", "target_dag_id"], - set_={"created_at": stmt.excluded.created_at}, - where=(AssetDagRunQueue.created_at < stmt.excluded.created_at), + session.execute( + insert(AssetDagRunQueue).on_conflict_do_nothing( + index_elements=["target_dag_id", "asset_event_id"] + ), + values, ) - session.execute(update_stmt, values) def resolve_asset_manager() -> AssetManager: diff --git a/airflow-core/src/airflow/cli/commands/asset_command.py b/airflow-core/src/airflow/cli/commands/asset_command.py index c5ea47d10d2e4..090525d59854f 100644 --- a/airflow-core/src/airflow/cli/commands/asset_command.py +++ b/airflow-core/src/airflow/cli/commands/asset_command.py @@ -107,7 +107,7 @@ def _detail_asset(args, *, session: Session) -> BaseModel: return AssetResponse.model_validate(asset) -@deprecated_for_airflowctl("airflowctl assets get / airflowctl assets get-by-alias") +@deprecated_for_airflowctl("airflowctl assets get / airflowctl assets get-alias") @cli_utils.action_cli @provide_session def asset_details(args, *, session: Session = NEW_SESSION) -> None: diff --git a/airflow-core/src/airflow/cli/commands/connection_command.py b/airflow-core/src/airflow/cli/commands/connection_command.py index bd5e736935bfb..48d90f15ad8b6 100644 --- a/airflow-core/src/airflow/cli/commands/connection_command.py +++ b/airflow-core/src/airflow/cli/commands/connection_command.py @@ -29,6 +29,7 @@ from sqlalchemy import select from sqlalchemy.orm import exc +from airflow._shared.secrets_masker import redact from airflow.cli.simple_table import AirflowConsole from airflow.cli.utils import ( SENSITIVE_PLACEHOLDER, @@ -365,22 +366,19 @@ def connections_add(args): with create_session() as session: if not session.scalar(select(Connection).where(Connection.conn_id == new_conn.conn_id).limit(1)): session.add(new_conn) - msg = "Successfully added `conn_id`={conn_id} : {uri}" - msg = msg.format( - conn_id=new_conn.conn_id, - uri=args.conn_uri - or urlunsplit( - ( - new_conn.conn_type, - f"{new_conn.login or ''}:{'******' if new_conn.password else ''}" - f"@{new_conn.host or ''}:{new_conn.port or ''}", - new_conn.schema or "", - "", - "", - ) - ), + print(f"Successfully added `conn_id`={new_conn.conn_id}") + AirflowConsole().print_as( + data=[new_conn], + output="table", + mapper=lambda conn: { + "conn_id": conn.conn_id, + "conn_type": conn.conn_type, + "host": conn.host, + "login": conn.login, + "port": conn.port, + "extra": redact(conn.extra_dejson), + }, ) - print(msg) else: msg = f"A connection with `conn_id`={new_conn.conn_id} already exists." raise SystemExit(msg) diff --git a/airflow-core/src/airflow/cli/commands/jobs_command.py b/airflow-core/src/airflow/cli/commands/jobs_command.py index 194d8720db2f1..57ab4a1df7b61 100644 --- a/airflow-core/src/airflow/cli/commands/jobs_command.py +++ b/airflow-core/src/airflow/cli/commands/jobs_command.py @@ -20,6 +20,7 @@ from sqlalchemy import select +from airflow.cli.utils import deprecated_for_airflowctl from airflow.jobs.job import Job, JobState from airflow.utils.net import get_hostname from airflow.utils.providers_configuration_loader import providers_configuration_loaded @@ -29,6 +30,7 @@ from sqlalchemy.orm import Session +@deprecated_for_airflowctl("airflowctl jobs check") @providers_configuration_loaded @provide_session def check(args, *, session: Session = NEW_SESSION) -> None: diff --git a/airflow-core/src/airflow/configuration.py b/airflow-core/src/airflow/configuration.py index 4eb92054ceab9..f9dd0af27b1db 100644 --- a/airflow-core/src/airflow/configuration.py +++ b/airflow-core/src/airflow/configuration.py @@ -58,8 +58,6 @@ warnings.filterwarnings(action="default", category=DeprecationWarning, module="airflow") warnings.filterwarnings(action="default", category=PendingDeprecationWarning, module="airflow") -_SQLITE3_VERSION_PATTERN = re.compile(r"(?P^\d+(?:\.\d+)*)\D?.*$") - ConfigType = str | int | float | bool ConfigOptionsDictType = dict[str, ConfigType] ConfigSectionSourcesType = dict[str, str | tuple[str, str]] @@ -99,13 +97,6 @@ def add_default_update(self, section: str, option: str, new_default: str) -> Non self.default_updates[(section, option)] = new_default -def _parse_sqlite_version(s: str) -> tuple[int, ...]: - match = _SQLITE3_VERSION_PATTERN.match(s) - if match is None: - return () - return tuple(int(p) for p in match.group("version").split(".")) - - @overload def expand_env_var(env_var: None) -> None: ... @@ -426,7 +417,7 @@ def _validate_sqlite3_version(self): import sqlite3 min_sqlite_version = (3, 15, 0) - if _parse_sqlite_version(sqlite3.sqlite_version) >= min_sqlite_version: + if sqlite3.sqlite_version_info >= min_sqlite_version: return from airflow.utils.docs import get_docs_url diff --git a/airflow-core/src/airflow/dag_processing/dagbag.py b/airflow-core/src/airflow/dag_processing/dagbag.py index 3bf82804bffe1..76de19240fad5 100644 --- a/airflow-core/src/airflow/dag_processing/dagbag.py +++ b/airflow-core/src/airflow/dag_processing/dagbag.py @@ -151,12 +151,12 @@ def _validate_executor_fields(dag: DAG, bundle_name: str | None = None) -> None: f"Task '{task.task_id}' specifies executor '{task.executor}', which is not available " f"for team '{dag_team_name}' (the team associated with DAG '{dag.dag_id}') or as a global executor. " f"Make sure '{task.executor}' is configured for team '{dag_team_name}' or globally in your " - "[core] executors configuration, or update the task's executor to use one of the " + "[core] executor configuration, or update the task's executor to use one of the " f"configured executors for team '{dag_team_name}' or available global executors." ) raise UnknownExecutorException( f"Task '{task.task_id}' specifies executor '{task.executor}', which is not available. " - "Make sure it is listed in your [core] executors configuration, or update the task's " + "Make sure it is listed in your [core] executor configuration, or update the task's " "executor to use one of the configured executors." ) diff --git a/airflow-core/src/airflow/dag_processing/manager.py b/airflow-core/src/airflow/dag_processing/manager.py index df0f8f8748135..e8a8a2c50366e 100644 --- a/airflow-core/src/airflow/dag_processing/manager.py +++ b/airflow-core/src/airflow/dag_processing/manager.py @@ -56,7 +56,6 @@ from airflow.dag_processing.bundles.manager import DagBundlesManager from airflow.dag_processing.collection import update_dag_parsing_results_in_db from airflow.dag_processing.processor import DagFileParsingResult, DagFileProcessorProcess -from airflow.exceptions import AirflowException from airflow.models.asset import remove_references_to_deleted_dags from airflow.models.dag import DagModel from airflow.models.dagbag import DagPriorityParsingRequest @@ -506,7 +505,11 @@ def deactivate_stale_dags( # When the Dag's last_parsed_time is more than the stale_dag_threshold older than the # Dag file's last_finish_time, the Dag is considered stale as has apparently been removed from the file, # This is especially relevant for Dag files that generate Dags in a dynamic manner. - file_info = DagFileInfo(rel_path=Path(dag.relative_fileloc), bundle_name=dag.bundle_name) + rel_path = Path(dag.relative_fileloc) + file_info = DagFileInfo(rel_path=rel_path, bundle_name=dag.bundle_name) + if file_info not in last_parsed: + # Zip-packaged dags are keyed by the archive path, not the inner file, so try the parent as well + file_info = DagFileInfo(rel_path=rel_path.parent, bundle_name=dag.bundle_name) if last_finish_time := last_parsed.get(file_info, None): if dag.last_parsed_time + timedelta(seconds=self.stale_dag_threshold) < last_finish_time: self.log.info( @@ -849,7 +852,7 @@ def _refresh_dag_bundles(self, known_files: dict[str, set[DagFileInfo]]): try: bundle.initialize() any_refreshed = True - except AirflowException as e: + except Exception as e: self.log.exception("Error initializing bundle %s: %s", bundle.name, e) continue try: diff --git a/airflow-core/src/airflow/jobs/scheduler_job_runner.py b/airflow-core/src/airflow/jobs/scheduler_job_runner.py index b6a29995d5937..3f2c7c8a7736d 100644 --- a/airflow-core/src/airflow/jobs/scheduler_job_runner.py +++ b/airflow-core/src/airflow/jobs/scheduler_job_runner.py @@ -28,7 +28,7 @@ from collections import Counter, defaultdict, deque from collections.abc import Callable, Collection, Iterable, Iterator from contextlib import ExitStack -from datetime import date, datetime, timedelta +from datetime import datetime, timedelta from functools import lru_cache, partial from itertools import groupby from typing import TYPE_CHECKING, Any, cast @@ -135,10 +135,10 @@ if TYPE_CHECKING: from types import FrameType - from pendulum.datetime import DateTime from sqlalchemy.engine import CursorResult from sqlalchemy.orm import Session from sqlalchemy.orm.interfaces import LoaderOption + from sqlalchemy.sql.elements import ColumnElement from sqlalchemy.sql.selectable import Subquery from airflow._shared.logging.types import Logger @@ -458,6 +458,24 @@ def _get_team_names_for_dag_ids( # Ensure all requested dag_ids are in the result (with None for those not found) return {dag_id: self._dag_id_to_team_name.get(dag_id) for dag_id in dag_ids} + def _stamp_team_names(self, dag_runs: Collection[DagRun], session: Session) -> None: + """ + Stamp ``_team_name`` on each DagRun. + + Team names are resolved via ``_get_team_names_for_dag_ids``, which caches results in + ``self._dag_id_to_team_name`` for the duration of the current scheduler loop. In + practice this means the first call per loop issues one batched query; subsequent calls + for the same dag_ids are pure dict reads with no DB round-trip. + """ + if not self._multi_team: + return + if not dag_runs: + return + team_map = self._get_team_names_for_dag_ids({dr.dag_id for dr in dag_runs}, session) + for dr in dag_runs: + if team := team_map.get(dr.dag_id): + dr._team_name = team + def _get_workload_team_name(self, workload: SchedulerWorkload, session: Session) -> str | None: """ Resolve team name for a workload using the DAG > Bundle > Team relationship chain. @@ -1734,12 +1752,8 @@ def _update_dag_run_state_for_paused_dags(self, *, session: Session = NEW_SESSIO .group_by(DagRun) ) ) - if self._multi_team and paused_runs: - paused_dag_ids = {dr.dag_id for dr in paused_runs} - paused_team_mapping = self._get_team_names_for_dag_ids(paused_dag_ids, session) - for dr in paused_runs: - if team := paused_team_mapping.get(dr.dag_id): - dr._team_name = team + # Team name should be added before listeners are called in update_state() + self._stamp_team_names(paused_runs, session) for dag_run in paused_runs: dag = self.scheduler_dag_bag.get_dag_for_run(dag_run=dag_run, session=session) if dag is not None: @@ -1828,6 +1842,7 @@ def _run_scheduler_loop(self) -> None: timers.call_regular_interval( conf.getfloat("scheduler", "parsing_cleanup_interval"), self._remove_unreferenced_triggers, + non_fatal=True, ) if any(x.is_local for x in self.executors): @@ -1845,6 +1860,7 @@ def _run_scheduler_loop(self) -> None: timers.call_regular_interval( delay=conf.getfloat("connection_test", "reaper_interval", fallback=30.0), action=self._reap_stale_connection_tests, + non_fatal=True, ) idle_count = 0 @@ -1996,12 +2012,8 @@ def _do_scheduling(self, session: Session) -> int: ) ) - if self._multi_team and dag_runs: - unique_dag_ids = {dr.dag_id for dr in dag_runs} - dr_team_mapping = self._get_team_names_for_dag_ids(unique_dag_ids, session) - for dr in dag_runs: - if team := dr_team_mapping.get(dr.dag_id): - dr._team_name = team + # Team name should be added before listeners are called in _schedule_all_dag_runs() + self._stamp_team_names(dag_runs, session) callback_tuples = self._schedule_all_dag_runs(guard, dag_runs, session) @@ -2641,9 +2653,7 @@ def _create_dag_runs_asset_triggered( queued_adrqs = session.scalars( with_row_locks( - select(AssetDagRunQueue) - .where(AssetDagRunQueue.target_dag_id == dag.dag_id) - .order_by(AssetDagRunQueue.created_at.desc()), + select(AssetDagRunQueue).where(AssetDagRunQueue.target_dag_id == dag.dag_id), of=AssetDagRunQueue, skip_locked=True, key_share=False, @@ -2658,53 +2668,31 @@ def _create_dag_runs_asset_triggered( ) continue - triggered_date: DateTime = timezone.coerce_datetime(queued_adrqs[0].created_at) - self.log.debug( - "Creating asset-triggered DagRun for '%s': %d queued assets, triggered_date=%s", - dag.dag_id, - len(queued_adrqs), - triggered_date, - ) - cte = ( - select(func.max(DagRun.run_after).label("previous_dag_run_run_after")) - .where( - DagRun.dag_id == dag.dag_id, - DagRun.run_type == DagRunType.ASSET_TRIGGERED, - DagRun.run_after < triggered_date, - ) - .cte() - ) - - # A first asset-triggered run has no previous run to floor the event window. With - # catchup off, floor it at when the Dag started scheduling on its assets so the - # backlog is skipped; with catchup on, only date.min applies and the backlog replays. - event_window_floor: list[Any] = [cte.c.previous_dag_run_run_after] - if not dag.catchup: - event_window_floor.append( - select(func.min(DagScheduleAssetReference.created_at)) - .where(DagScheduleAssetReference.dag_id == dag.dag_id) - .scalar_subquery() + referenced_event_ids = {adrq.asset_event_id for adrq in queued_adrqs} + event_predicate: ColumnElement[bool] = AssetEvent.id.in_(referenced_event_ids) + if dag.catchup: + # With catchup on, also consume events recorded before the Dag started + # scheduling on its assets/aliases, not just those with a queue row. (With catchup + # off only queued events are consumed.) The not-consumed filter below dedupes + # across runs, so no event window is needed. + event_predicate = or_( + event_predicate, + AssetEvent.asset_id.in_( + select(DagScheduleAssetReference.asset_id).where( + DagScheduleAssetReference.dag_id == dag.dag_id + ) + ), + AssetEvent.source_aliases.any( + AssetAliasModel.scheduled_dags.any( + DagScheduleAssetAliasReference.dag_id == dag.dag_id + ) + ), ) - event_window_floor.append(date.min) - asset_events = list( session.scalars( select(AssetEvent) .where( - or_( - AssetEvent.asset_id.in_( - select(DagScheduleAssetReference.asset_id).where( - DagScheduleAssetReference.dag_id == dag.dag_id - ) - ), - AssetEvent.source_aliases.any( - AssetAliasModel.scheduled_dags.any( - DagScheduleAssetAliasReference.dag_id == dag.dag_id - ) - ), - ), - AssetEvent.timestamp > func.coalesce(*event_window_floor), - AssetEvent.timestamp <= triggered_date, + event_predicate, ~( select(association_table.c.event_id) .join(DagRun, DagRun.id == association_table.c.dag_run_id) @@ -2719,6 +2707,13 @@ def _create_dag_runs_asset_triggered( ) ) if asset_events: + triggered_date = timezone.coerce_datetime(max(event.timestamp for event in asset_events)) + self.log.debug( + "Creating asset-triggered DagRun for '%s': %d queued assets, triggered_date=%s", + dag.dag_id, + len(queued_adrqs), + triggered_date, + ) dag_run = dag.create_dagrun( run_id=DagRun.generate_run_id( run_type=DagRunType.ASSET_TRIGGERED, logical_date=None, run_after=triggered_date @@ -2747,19 +2742,19 @@ def _create_dag_runs_asset_triggered( ) else: self.log.info( - "No DagRun created for '%s' at '%s' - asset events already consumed or none found", + "No DagRun created for '%s' - asset events already consumed or none found", dag.dag_id, - triggered_date, ) # Always delete ADRQ rows for this batch to prevent stale entries accumulating, # including when all events were already consumed by a concurrent DagRun. - adrq_pks = [(record.asset_id, record.target_dag_id) for record in queued_adrqs] result = cast( "CursorResult", session.execute( delete(AssetDagRunQueue).where( - tuple_(AssetDagRunQueue.asset_id, AssetDagRunQueue.target_dag_id).in_(adrq_pks), - AssetDagRunQueue.created_at <= triggered_date, + tuple_( + AssetDagRunQueue.target_dag_id, + AssetDagRunQueue.asset_event_id, + ).in_((adrq.target_dag_id, adrq.asset_event_id) for adrq in queued_adrqs) ) ), ) @@ -2844,6 +2839,9 @@ def _update_state(dag: SerializedDAG, dag_run: DagRun): partial(self.scheduler_dag_bag.get_dag_for_run, session=session) ) + # Team name should be added before listeners are called in notify_dagrun_state_changed() + self._stamp_team_names(dag_runs, session) + for dag_run in dag_runs: dag_id = dag_run.dag_id run_id = dag_run.run_id @@ -2981,6 +2979,8 @@ def _schedule_dag_run( execute=False, ) + # Team name should be added before listeners are called in notify_dagrun_state_changed() + self._stamp_team_names([dag_run], session) dag_run.notify_dagrun_state_changed(msg="timed_out") if dag_run.end_date and dag_run.start_date: duration = dag_run.end_date - dag_run.start_date @@ -3528,8 +3528,10 @@ def check_awaiting_input_timeouts( num_resolved = 0 num_failed = 0 + num_unresumable = 0 for ti in timed_out_tis: hitl_detail = ti.hitl_detail + resuming = True if hitl_detail is not None and hitl_detail.responded_at is not None: # A response landed just before the deadline; resume with it. handle_event_submit( @@ -3537,7 +3539,6 @@ def check_awaiting_input_timeouts( task_instance=ti, session=session, ) - num_resolved += 1 elif hitl_detail is not None and hitl_detail.defaults is not None: # Apply the configured defaults as the response, then resume to success. hitl_detail.chosen_options = list(hitl_detail.defaults) @@ -3553,7 +3554,6 @@ def check_awaiting_input_timeouts( task_instance=ti, session=session, ) - num_resolved += 1 else: # No defaults and no response: resume into execute_complete with a timeout # failure event so the operator raises HITLTimeoutError (matching the old @@ -3569,16 +3569,29 @@ def check_awaiting_input_timeouts( task_instance=ti, session=session, ) + resuming = False + + # ``handle_event_submit`` routes a task instance it could not process to + # ``__fail__`` instead of resuming it. That is neither of the outcomes the + # branches above intended, so it is counted on its own rather than being + # reported as resolved. + if ti.next_method == TRIGGER_FAIL_REPR: + num_unresumable += 1 + elif resuming: + num_resolved += 1 + else: num_failed += 1 # Flush within the retry block so both branches persist consistently (the defaults # branch already flushes via handle_event_submit; the fail branch relies on this). session.flush() - if num_resolved or num_failed: + if num_resolved or num_failed or num_unresumable: self.log.info( - "AWAITING_INPUT timeout sweep: %i resolved (response/defaults), %i failed", + "AWAITING_INPUT timeout sweep: %i resolved (response/defaults), %i failed, " + "%i could not be resumed", num_resolved, num_failed, + num_unresumable, ) # [START find_and_purge_task_instances_without_heartbeats] diff --git a/airflow-core/src/airflow/migrations/versions/0128_3_4_0_add_asset_event_id_to_adrq.py b/airflow-core/src/airflow/migrations/versions/0128_3_4_0_add_asset_event_id_to_adrq.py new file mode 100644 index 0000000000000..ff2a228218112 --- /dev/null +++ b/airflow-core/src/airflow/migrations/versions/0128_3_4_0_add_asset_event_id_to_adrq.py @@ -0,0 +1,205 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Reference the asset event from asset_dag_run_queue (consume-by-reference). + +Table ``asset_dag_run_queue`` gets a new column ``asset_event_id``, and the +primary key is made ``(target_dag_id, asset_event_id)``, so the scheduler +consumes queued asset events by reference instead of by a ``created_at`` time +window. ``asset_id`` is kept as a denormalized column. + +Existing rows are coalesced (one per ``(asset_id, target_dag_id)``) and carry no +event reference. Rather than dropping them (which would silently skip pending +asset-triggered Dag runs), the pre-migration scheduler's own consumption window is +replayed per dag to rebuild per-event rows: + + triggered_date = + MAX(asset_dag_run_queue.created_at) (per dag) + floor = COALESCE( + MAX(dag_run.run_after) for asset-triggered runs of the dag with + run_after < triggered_date, (per dag) + dag_schedule_asset_reference.created_at, (per dag + asset) + ) + contributing events = + the queued asset's events with + floor < asset_event.timestamp <= triggered_date + +The expansion is staged in a side table, the queue is cleared, the (now empty) +table is reshaped, and the staged rows are inserted back. + +Revision ID: b2f1a9c7d4e0 +Revises: 7a98f1b7dbd3 +Create Date: 2026-08-03 12:00:00.000000 +""" + +from __future__ import annotations + +from datetime import datetime, timezone +from textwrap import dedent + +import sqlalchemy as sa +from alembic import context, op + +# revision identifiers, used by Alembic. +revision = "b2f1a9c7d4e0" +down_revision = "7a98f1b7dbd3" +branch_labels = None +depends_on = None +airflow_version = "3.4.0" + +_STAGING = "_adrq_migration_staging" + +_STAGE_SQL = f""" +CREATE TABLE {_STAGING} AS +SELECT DISTINCT + adrq.target_dag_id AS target_dag_id, + adrq.asset_id AS asset_id, + ae.id AS asset_event_id +FROM asset_dag_run_queue adrq +JOIN ( + SELECT t.target_dag_id, + t.triggered_date, + ( + SELECT MAX(dr.run_after) + FROM dag_run dr + WHERE dr.dag_id = t.target_dag_id + AND dr.run_type = 'asset_triggered' + AND dr.run_after < t.triggered_date + ) AS floor_date + FROM ( + SELECT target_dag_id, MAX(created_at) AS triggered_date + FROM asset_dag_run_queue + GROUP BY target_dag_id + ) t +) td ON td.target_dag_id = adrq.target_dag_id +LEFT JOIN dag_schedule_asset_reference dsar + ON dsar.dag_id = adrq.target_dag_id + AND dsar.asset_id = adrq.asset_id +JOIN asset_event ae + ON ae.asset_id = adrq.asset_id + AND ae.timestamp <= td.triggered_date + AND ae.timestamp > COALESCE(td.floor_date, dsar.created_at, :floor_min) +""" + + +def upgrade(): + """Directly reference the asset event from asset_dag_run_queue.""" + # 1. Add the reference column, nullable for now; it is made NOT NULL after the rebuild. + op.add_column("asset_dag_run_queue", sa.Column("asset_event_id", sa.Integer(), nullable=True)) + + # 2. Stage the faithful per-event expansion, then clear the coalesced rows. + if context.is_offline_mode(): + print( + dedent(""" + ------------ + -- WARNING: asset_dag_run_queue cannot be rebuilt in offline mode; + -- any pending (unprocessed) queued asset events will be dropped. + ------------ + """) + ) + op.execute("DELETE FROM asset_dag_run_queue") + else: + conn = op.get_bind() + floor_min = datetime(1970, 1, 1, tzinfo=timezone.utc) + conn.execute(sa.text(_STAGE_SQL), {"floor_min": floor_min}) + conn.execute(sa.text("DELETE FROM asset_dag_run_queue")) + + # 3. Make the reference NOT NULL, move the PK, adjust FKs and indexes. + with op.batch_alter_table("asset_dag_run_queue") as batch_op: + batch_op.alter_column("asset_event_id", existing_type=sa.Integer(), nullable=False) + batch_op.drop_constraint("adrq_asset_fkey", type_="foreignkey") + batch_op.drop_constraint("assetdagrunqueue_pkey", type_="primary") + batch_op.create_primary_key("assetdagrunqueue_pkey", ["target_dag_id", "asset_event_id"]) + batch_op.create_index("idx_adrq_asset_id", ["asset_id"]) + batch_op.create_foreign_key("adrq_asset_fkey", "asset", ["asset_id"], ["id"], ondelete="CASCADE") + batch_op.create_foreign_key( + "adrq_asset_event_fkey", "asset_event", ["asset_event_id"], ["id"], ondelete="CASCADE" + ) + batch_op.drop_index("idx_asset_dag_run_queue_target_dag_id") + + # 4. Repopulate the per-event rows from staging. + if not context.is_offline_mode(): + conn = op.get_bind() + now = datetime.now(timezone.utc) + conn.execute( + sa.text( + f""" + INSERT INTO asset_dag_run_queue (asset_id, target_dag_id, asset_event_id, created_at) + SELECT asset_id, target_dag_id, asset_event_id, :now + FROM {_STAGING} + """ + ), + {"now": now}, + ) + op.drop_table(_STAGING) + + +def downgrade(): + """Revert reference to asset_id from asset_dag_run_queue.""" + # 1. Rebuild the coalesced rows: collapse per-event rows to one row per + # (asset_id, target_dag_id) (created_at = max referenced event timestamp), then clear. + if context.is_offline_mode(): + print( + dedent(""" + ------------ + -- WARNING: asset_dag_run_queue cannot be rebuilt in offline mode; + -- any pending (unprocessed) queued asset events will be dropped. + ------------ + """) + ) + op.execute("DELETE FROM asset_dag_run_queue") + else: + conn = op.get_bind() + conn.execute( + sa.text( + f""" + CREATE TABLE {_STAGING} AS + SELECT adrq.asset_id AS asset_id, + adrq.target_dag_id AS target_dag_id, + MAX(ae.timestamp) AS created_at + FROM asset_dag_run_queue adrq + JOIN asset_event ae ON ae.id = adrq.asset_event_id + GROUP BY adrq.asset_id, adrq.target_dag_id + """ + ) + ) + conn.execute(sa.text("DELETE FROM asset_dag_run_queue")) + + # 2. Drop the reference + FK, restore the old primary key and indexes. + with op.batch_alter_table("asset_dag_run_queue") as batch_op: + batch_op.drop_constraint("adrq_asset_event_fkey", type_="foreignkey") + batch_op.create_index("idx_asset_dag_run_queue_target_dag_id", ["target_dag_id"]) + batch_op.drop_constraint("assetdagrunqueue_pkey", type_="primary") + batch_op.create_primary_key("assetdagrunqueue_pkey", ["asset_id", "target_dag_id"]) + batch_op.drop_index("idx_adrq_asset_id") + batch_op.drop_column("asset_event_id") + + # 3. Repopulate the coalesced rows. + if not context.is_offline_mode(): + conn = op.get_bind() + conn.execute( + sa.text( + f""" + INSERT INTO asset_dag_run_queue (asset_id, target_dag_id, created_at) + SELECT asset_id, target_dag_id, created_at + FROM {_STAGING} + """ + ) + ) + op.drop_table(_STAGING) diff --git a/airflow-core/src/airflow/migrations/versions/0129_3_4_0_add_indexes_on_serialized_dag_and_dag_.py b/airflow-core/src/airflow/migrations/versions/0129_3_4_0_add_indexes_on_serialized_dag_and_dag_.py new file mode 100644 index 0000000000000..5adb80033dd50 --- /dev/null +++ b/airflow-core/src/airflow/migrations/versions/0129_3_4_0_add_indexes_on_serialized_dag_and_dag_.py @@ -0,0 +1,55 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Add indexes on serialized_dag and dag_code. + +Revision ID: 3c525f44bea8 +Revises: b2f1a9c7d4e0 +Create Date: 2026-07-04 18:18:11.140047 + +""" + +from __future__ import annotations + +from alembic import op + +# revision identifiers, used by Alembic. +revision = "3c525f44bea8" +down_revision = "b2f1a9c7d4e0" +branch_labels = None +depends_on = None +airflow_version = "3.4.0" + + +def upgrade(): + """Apply Add indexes on serialized_dag and dag_code.""" + with op.batch_alter_table("dag_code", schema=None) as batch_op: + batch_op.create_index("idx_dag_code_dag_id_last_updated", ["dag_id", "last_updated"], unique=False) + + with op.batch_alter_table("serialized_dag", schema=None) as batch_op: + batch_op.create_index("idx_serialized_dag_dag_id_created_at", ["dag_id", "created_at"], unique=False) + + +def downgrade(): + """Unapply Add indexes on serialized_dag and dag_code.""" + with op.batch_alter_table("serialized_dag", schema=None) as batch_op: + batch_op.drop_index("idx_serialized_dag_dag_id_created_at") + + with op.batch_alter_table("dag_code", schema=None) as batch_op: + batch_op.drop_index("idx_dag_code_dag_id_last_updated") diff --git a/airflow-core/src/airflow/models/asset.py b/airflow-core/src/airflow/models/asset.py index 7aafddb3b9490..750aac8d2b73b 100644 --- a/airflow-core/src/airflow/models/asset.py +++ b/airflow-core/src/airflow/models/asset.py @@ -749,33 +749,40 @@ def __repr__(self): class AssetDagRunQueue(Base): """Model for storing asset events that need processing.""" - asset_id: Mapped[int] = mapped_column(Integer, primary_key=True, nullable=False) target_dag_id: Mapped[str] = mapped_column(StringID(), primary_key=True, nullable=False) + asset_event_id: Mapped[int] = mapped_column(Integer, primary_key=True, nullable=False) + asset_id: Mapped[int] = mapped_column(Integer, nullable=False) created_at: Mapped[datetime] = mapped_column(UtcDateTime, default=timezone.utcnow, nullable=False) asset: Mapped[AssetModel] = relationship("AssetModel", viewonly=True) dag_model: Mapped[DagModel] = relationship("DagModel", viewonly=True) __tablename__ = "asset_dag_run_queue" __table_args__ = ( - PrimaryKeyConstraint(asset_id, target_dag_id, name="assetdagrunqueue_pkey"), + PrimaryKeyConstraint(target_dag_id, asset_event_id, name="assetdagrunqueue_pkey"), ForeignKeyConstraint( (asset_id,), ["asset.id"], name="adrq_asset_fkey", ondelete="CASCADE", ), + ForeignKeyConstraint( + (asset_event_id,), + ["asset_event.id"], + name="adrq_asset_event_fkey", + ondelete="CASCADE", + ), ForeignKeyConstraint( (target_dag_id,), ["dag.dag_id"], name="adrq_dag_fkey", ondelete="CASCADE", ), - Index("idx_asset_dag_run_queue_target_dag_id", target_dag_id), + Index("idx_adrq_asset_id", asset_id), ) def __eq__(self, other: object) -> bool: if isinstance(other, self.__class__): - return self.asset_id == other.asset_id and self.target_dag_id == other.target_dag_id + return self.target_dag_id == other.target_dag_id and self.asset_event_id == other.asset_event_id return NotImplemented def __hash__(self): diff --git a/airflow-core/src/airflow/models/dag.py b/airflow-core/src/airflow/models/dag.py index 29407e53bb0a0..d6dfab06a2707 100644 --- a/airflow-core/src/airflow/models/dag.py +++ b/airflow-core/src/airflow/models/dag.py @@ -36,7 +36,6 @@ Integer, String, Text, - and_, case, func, or_, @@ -60,7 +59,7 @@ from airflow.assets.evaluation import AssetEvaluator from airflow.configuration import conf as airflow_conf from airflow.exceptions import AirflowException -from airflow.models.asset import AssetDagRunQueue, AssetModel +from airflow.models.asset import AssetDagRunQueue from airflow.models.base import Base, StringID from airflow.models.dagbundle import DagBundleModel from airflow.models.dagrun import DagRun @@ -263,47 +262,6 @@ def get_last_dagrun(dag_id: str, session: Session, include_manually_triggered: b return session.scalar(query.limit(1)) -def get_asset_triggered_next_run_info( - dag_ids: list[str], *, session: Session -) -> dict[str, dict[str, int | str]]: - """ - Get next run info for a list of dag_ids. - - Given a list of dag_ids, get string representing how close any that are asset triggered are to - their next run, e.g. "1 of 2 assets updated". - """ - from airflow.models.asset import AssetDagRunQueue as ADRQ, DagScheduleAssetReference - - return { - x.dag_id: { - "uri": x.uri, - "ready": x.ready, - "total": x.total, - } - for x in session.execute( - select( - DagScheduleAssetReference.dag_id, - # This is a dirty hack to workaround group by requiring an aggregate, - # since grouping by asset is not what we want to do here...but it works - case((func.count() == 1, func.max(AssetModel.uri)), else_="").label("uri"), - func.count().label("total"), - func.sum(case((ADRQ.target_dag_id.is_not(None), 1), else_=0)).label("ready"), - ) - .join( - ADRQ, - and_( - ADRQ.asset_id == DagScheduleAssetReference.asset_id, - ADRQ.target_dag_id == DagScheduleAssetReference.dag_id, - ), - isouter=True, - ) - .join(AssetModel, AssetModel.id == DagScheduleAssetReference.asset_id) - .group_by(DagScheduleAssetReference.dag_id) - .where(DagScheduleAssetReference.dag_id.in_(dag_ids)) - ).all() - } - - class DagTag(Base): """A tag name per dag, to allow quick filtering in the DAG view.""" @@ -855,17 +813,6 @@ def calculate_dagrun_date_fields( next_dagrun_partition_date=str(self.next_dagrun_partition_date), ) - @provide_session - def get_asset_triggered_next_run_info( - self, *, session: Session = NEW_SESSION - ) -> dict[str, int | str] | None: - if self.asset_expression is None: - return None - - # When an asset alias does not resolve into assets, get_asset_triggered_next_run_info returns - # an empty dict as there's no asset info to get. This method should thus return None. - return get_asset_triggered_next_run_info([self.dag_id], session=session).get(self.dag_id, None) - @staticmethod @cached(_team_name_cache, key=lambda dag_id, **_: dag_id, lock=_team_name_cache_lock) @provide_session diff --git a/airflow-core/src/airflow/models/dagcode.py b/airflow-core/src/airflow/models/dagcode.py index d591ddb25b7b3..c693f0384cecb 100644 --- a/airflow-core/src/airflow/models/dagcode.py +++ b/airflow-core/src/airflow/models/dagcode.py @@ -23,7 +23,7 @@ import sqlalchemy as sa import uuid6 -from sqlalchemy import ForeignKey, String, Text, select +from sqlalchemy import ForeignKey, Index, String, Text, select from sqlalchemy.dialects.mysql import MEDIUMTEXT from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.sql.expression import literal @@ -70,6 +70,7 @@ class DagCode(Base): sa.Uuid(), ForeignKey("dag_version.id", ondelete="CASCADE"), nullable=False, unique=True ) dag_version = relationship("DagVersion", back_populates="dag_code", uselist=False) + __table_args__ = (Index("idx_dag_code_dag_id_last_updated", dag_id, last_updated),) def __init__(self, dag_version, full_filepath: str, source_code: str | None = None): self.dag_version = dag_version diff --git a/airflow-core/src/airflow/models/serialized_dag.py b/airflow-core/src/airflow/models/serialized_dag.py index f3f5b78309680..fa7d80335a304 100644 --- a/airflow-core/src/airflow/models/serialized_dag.py +++ b/airflow-core/src/airflow/models/serialized_dag.py @@ -27,7 +27,7 @@ from uuid import UUID import uuid6 -from sqlalchemy import JSON, ForeignKey, LargeBinary, String, Uuid, exists, select, tuple_, update +from sqlalchemy import JSON, ForeignKey, Index, LargeBinary, String, Uuid, exists, select, tuple_, update from sqlalchemy.dialects.postgresql import ARRAY, JSONB from sqlalchemy.orm import Mapped, backref, foreign, mapped_column, relationship from sqlalchemy.sql.expression import func, literal @@ -344,6 +344,7 @@ class SerializedDagModel(Base): ) load_op_links = True + __table_args__ = (Index("idx_serialized_dag_dag_id_created_at", dag_id, created_at),) def __init__(self, dag: LazyDeserializedDAG) -> None: self.dag_id = dag.dag_id diff --git a/airflow-core/src/airflow/models/taskinstance.py b/airflow-core/src/airflow/models/taskinstance.py index 0dd10507fc863..dbcb311afe447 100644 --- a/airflow-core/src/airflow/models/taskinstance.py +++ b/airflow-core/src/airflow/models/taskinstance.py @@ -24,7 +24,7 @@ import math import warnings from collections import defaultdict -from collections.abc import Collection, Iterable +from collections.abc import Callable, Collection, Iterable, Sequence from datetime import datetime, timedelta from typing import TYPE_CHECKING, Any, NamedTuple from urllib.parse import quote @@ -1546,14 +1546,14 @@ def register_asset_changes_in_db( outlet_events: list[dict[str, Any]], *, session: Session = NEW_SESSION, - ) -> None: + ) -> Sequence[Callable[[], None]]: # Fast path: a task with no outlets and no outlet events has nothing to # register. Returning early avoids the AssetModel lookup below (which # would run with empty IN () clauses) and all downstream work. This is # the common case -- most tasks declare no outlets -- and it sits on the # task-success path that gates scheduling the next task. if not task_outlets and not outlet_events: - return + return () from airflow.serialization.definitions.assets import ( SerializedAsset, @@ -1579,6 +1579,7 @@ def register_asset_changes_in_db( dag_run_partition_key = ti.dag_run.partition_key dag_run_partition_date = ti.dag_run.partition_date + callback_sink: list[Callable[[], None]] = [] asset_keys = { SerializedAssetUniqueKey(o.name, o.uri) for o in task_outlets @@ -1614,6 +1615,7 @@ def _register(am: AssetModel, key: SerializedAssetUniqueKey) -> None: extra=None, partition_key=dag_run_partition_key, partition_date=dag_run_partition_date, + callback_sink=callback_sink, session=session, ) return @@ -1641,6 +1643,7 @@ def _register(am: AssetModel, key: SerializedAssetUniqueKey) -> None: extra=payload.extra, partition_key=effective_pk, partition_date=payload_partition_date, + callback_sink=callback_sink, session=session, ) @@ -1725,6 +1728,7 @@ def _asset_event_extras_from_aliases() -> dict[tuple[SerializedAssetUniqueKey, s extra=asset_event_extra, partition_key=dag_run_partition_key, partition_date=dag_run_partition_date, + callback_sink=callback_sink, session=session, ) if event is None: @@ -1738,9 +1742,12 @@ def _asset_event_extras_from_aliases() -> dict[tuple[SerializedAssetUniqueKey, s extra=asset_event_extra, partition_key=dag_run_partition_key, partition_date=dag_run_partition_date, + callback_sink=callback_sink, session=session, ) + return callback_sink + @provide_session def update_rtif(self, rendered_fields, *, session: Session = NEW_SESSION): from airflow.models.renderedtifields import RenderedTaskInstanceFields diff --git a/airflow-core/src/airflow/models/trigger.py b/airflow-core/src/airflow/models/trigger.py index ef2adefd2eb08..a49d4bba2795f 100644 --- a/airflow-core/src/airflow/models/trigger.py +++ b/airflow-core/src/airflow/models/trigger.py @@ -500,6 +500,55 @@ def get_sorted_triggers( return result +def _decode_next_kwargs(next_kwargs_raw: Any) -> dict[str, Any]: + """ + Decode the stored ``next_kwargs`` of a task instance into a plain dict. + + Deserialize with serde first to provide a compat layer if there are mixed serialized + (BaseSerialisation and serde) data, which can happen if a deferred task resumes after upgrade. + + The result is checked here rather than assumed, so callers never have to trust the shape of + what comes back out of the stored payload. + + :raise ValueError: The payload did not decode to a dict. + :raise Exception: Whatever the two decoders raise on a payload they cannot read -- the stored + blob is arbitrary, so the set is open and callers have to treat it as such. + """ + from airflow.sdk.serde import deserialize + + try: + next_kwargs = deserialize(next_kwargs_raw) + except (ImportError, KeyError, AttributeError, TypeError): + from airflow.serialization.serialized_objects import BaseSerialization + + next_kwargs = BaseSerialization.deserialize(next_kwargs_raw) + + if not isinstance(next_kwargs, dict): + raise ValueError(f"next_kwargs decoded to {type(next_kwargs).__name__}, expected a dict") + return next_kwargs + + +def _fail_unresumable_task_instance( + task_instance: TaskInstance, reason: str, exc: BaseException, *, session: Session +) -> None: + """ + Route through ``__fail__`` so a worker fails the task normally instead of stranding it. + + Mirrors ``Trigger.submit_failure``: without this the task is left with no event to resume it. + Traceback goes into ``next_kwargs`` as a list -- the only channel reaching the task log -- + since ``format_exception`` returns it that way and the runtime joins it. + """ + task_instance.next_method = TRIGGER_FAIL_REPR + task_instance.next_kwargs = { + "error": reason, + "traceback": format_exception(type(exc), exc, exc.__traceback__), + } + task_instance.trigger_id = None + task_instance.state = TaskInstanceState.SCHEDULED + task_instance.scheduled_dttm = timezone.utcnow() + session.flush() + + @singledispatch def handle_event_submit(event: TriggerEvent, *, task_instance: TaskInstance, session: Session) -> None: """ @@ -509,33 +558,66 @@ def handle_event_submit(event: TriggerEvent, *, task_instance: TaskInstance, ses as well as its state to scheduled. It also adds the event's payload into the kwargs for the task. + A task instance whose stored kwargs cannot be decoded, or which the event payload cannot be + encoded into, is failed rather than resumed. This runs in the triggerer, the scheduler and the + API processes, each of which handles every waiting task instance in one pass, so a single + unusable payload must not be able to abort the caller. The triggerer had the worst of it: an + event whose submit raised was left unconfirmed and redelivered indefinitely. + + Failing the task instance is not free for every caller: a Human-in-the-loop response whose + ``params_input`` serde cannot encode is now recorded and discarded rather than rejected, which + the submitter cannot retry. That wants validating on the write side; tracked at + https://github.com/apache/airflow/issues/71036 + :param task_instance: The task instance to handle the submit event for. :param session: The session to be used for the database callback sink. """ - from airflow.sdk.serde import deserialize, serialize - from airflow.utils.state import TaskInstanceState + from airflow.sdk.serde import serialize next_kwargs_raw = task_instance.next_kwargs or {} - # deserialize first to provide a compat layer if there are mixed serialized (BaseSerialisation and serde) data - # which can happen if a deferred task resumes after upgrade + # Decoding and re-encoding fail for different reasons and are reported separately: blaming the + # stored kwargs for a payload the trigger just yielded would point the author at DB state that + # was never the problem. try: - next_kwargs = deserialize(next_kwargs_raw) - except (ImportError, KeyError, AttributeError, TypeError): - from airflow.serialization.serialized_objects import BaseSerialization - - next_kwargs = BaseSerialization.deserialize(next_kwargs_raw) + next_kwargs = _decode_next_kwargs(next_kwargs_raw) + except Exception as exc: + log.exception( + "Could not decode the stored next_kwargs of %s; failing it instead of resuming it", + task_instance, + ) + _fail_unresumable_task_instance( + task_instance, + "Could not resume the task: its stored next_kwargs could not be decoded " + f"({type(exc).__name__}: {exc})", + exc, + session=session, + ) + return # Add event to the plain dict, then serialize everything together so nested # non-primitive values get proper serde encoding. - if TYPE_CHECKING: - assert isinstance(next_kwargs, dict) next_kwargs["event"] = event.payload + try: + # Re-serialize using serde. The Execution API version converter + # (ModifyDeferredTaskKwargsToJsonValue) handles converting this to + # BaseSerialization format when serving old workers. + serialized_next_kwargs = serialize(next_kwargs) + except Exception as exc: + log.exception( + "Could not serialize the event payload for %s; failing it instead of resuming it", + task_instance, + ) + _fail_unresumable_task_instance( + task_instance, + f"Could not resume the task: the event payload could not be serialized " + f"({type(exc).__name__}: {exc})", + exc, + session=session, + ) + return - # Re-serialize using serde. The Execution API version converter - # (ModifyDeferredTaskKwargsToJsonValue) handles converting this to - # BaseSerialization format when serving old workers. - task_instance.next_kwargs = serialize(next_kwargs) + task_instance.next_kwargs = serialized_next_kwargs # Remove ourselves as its trigger task_instance.trigger_id = None @@ -560,13 +642,35 @@ def _(event: BaseTaskEndEvent, *, task_instance: TaskInstance, session: Session) from airflow.callbacks.database_callback_sink import DatabaseCallbackSink from airflow.utils.state import TaskInstanceState - # Mark the task with terminal state and prevent it from resuming on worker + # Prevent the task from resuming on a worker. task_instance.trigger_id = None - task_instance.set_state(event.task_instance_state, session=session) + + callback_type = event.task_instance_state + should_retry = False + + if event.task_instance_state == TaskInstanceState.FAILED: + # Load the serialized task so retry eligibility matches the normal task path. + try: + from airflow.models.dagbag import DBDagBag + + dag = DBDagBag().get_dag_for_run(dag_run=task_instance.dag_run, session=session) + if dag is not None: + task_instance.task = dag.get_task(task_instance.task_id) + should_retry = task_instance.is_eligible_to_retry() + except Exception: + log.exception( + "Could not load task for %s; failing terminally without retry routing", task_instance + ) + if should_retry: + callback_type = TaskInstanceState.UP_FOR_RETRY def _submit_callback_if_necessary() -> None: - """Submit a callback request if the task state is SUCCESS or FAILED.""" - if event.task_instance_state in (TaskInstanceState.SUCCESS, TaskInstanceState.FAILED): + """Submit a callback request if the task state is SUCCESS, FAILED, or UP_FOR_RETRY.""" + if callback_type in ( + TaskInstanceState.SUCCESS, + TaskInstanceState.FAILED, + TaskInstanceState.UP_FOR_RETRY, + ): if task_instance.dag_model.relative_fileloc is None: raise RuntimeError("relative_fileloc should not be None for a finished task") from airflow.models.dag_version import _resolve_version_data @@ -590,7 +694,7 @@ def _submit_callback_if_necessary() -> None: request = TaskCallbackRequest( filepath=task_instance.dag_model.relative_fileloc, ti=task_instance, - task_callback_type=event.task_instance_state, + task_callback_type=callback_type, bundle_name=bundle_name, bundle_version=bundle_version, version_data=version_data, @@ -603,10 +707,22 @@ def _submit_callback_if_necessary() -> None: def _push_xcoms_if_necessary() -> None: """Pushes XComs to the database if they are provided.""" - if event.xcoms: + if event.xcoms and callback_type != TaskInstanceState.UP_FOR_RETRY: for key, value in event.xcoms.items(): task_instance.xcom_push(key=key, value=value) + # Send the callback before mutating task state so it reflects the retry-vs-terminal + # decision derived above. _submit_callback_if_necessary() + + if should_retry: + task_instance.end_date = timezone.utcnow() + task_instance.set_duration() + task_instance.clear_next_method_args() + task_instance.prepare_db_for_next_try(session) + task_instance.state = TaskInstanceState.UP_FOR_RETRY + else: + task_instance.set_state(event.task_instance_state, session=session) + _push_xcoms_if_necessary() session.flush() diff --git a/airflow-core/src/airflow/provider.yaml.schema.json b/airflow-core/src/airflow/provider.yaml.schema.json index 0a700aa88d550..d1b0651c670f7 100644 --- a/airflow-core/src/airflow/provider.yaml.schema.json +++ b/airflow-core/src/airflow/provider.yaml.schema.json @@ -471,6 +471,14 @@ }, "additionalProperties": false }, + "external-services": { + "description": "Representative, non-exhaustive list of upstream provider/service names this connection type is known to reach over the network (e.g. 'Anthropic', 'AWS Bedrock', 'Ollama') -- examples, not an exhaustive compatibility matrix. Hooks that resolve the destination from a caller-supplied model identifier (as with pydantic-ai) can reach any service the model id names, so the list can never be complete. Not to be confused with the top-level 'integrations' key, which describes framework integrations and drives docs/logo/tag generation.", + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + }, "conn-fields": { "description": "Custom connection fields stored in Connection.extra JSON", "type": "object", @@ -557,12 +565,22 @@ "type": "array", "description": "Decorators to use with the TaskFlow API. Can be accessed by users via '@task.'", "items": { - "name": { - "type": "string" + "type": "object", + "properties": { + "name": { + "description": "Name the decorator is exposed under, following '@task.'", + "type": "string" + }, + "class-name": { + "description": "Class name that implements the decorator", + "type": "string" + } }, - "path": { - "type": "string" - } + "additionalProperties": false, + "required": [ + "name", + "class-name" + ] } }, "secrets-backends": { @@ -680,24 +698,29 @@ "type": "array", "description": "Plugins exposed by the provider", "items": { - "name": { - "type": "string" + "type": "object", + "properties": { + "name": { + "description": "Name of the plugin", + "type": "string" + }, + "plugin-class": { + "description": "Class name that implements the plugin", + "type": "string" + } }, - "plugin-class": { - "type": "string" - } + "additionalProperties": false, + "required": [ + "name", + "plugin-class" + ] } }, "queues": { "type": "array", - "description": "Message Queues exposed by the provider", + "description": "Message queue provider class names", "items": { - "name": { - "type": "string" - }, - "message-queue-class": { - "type": "string" - } + "type": "string" } }, "source-date-epoch": { diff --git a/airflow-core/src/airflow/provider_info.schema.json b/airflow-core/src/airflow/provider_info.schema.json index b3e9bf74b6284..4e6ddacebd1fb 100644 --- a/airflow-core/src/airflow/provider_info.schema.json +++ b/airflow-core/src/airflow/provider_info.schema.json @@ -432,11 +432,16 @@ "type": "array", "description": "Apply custom decorators to the TaskFlow API. Can be accessed by users via '@task.'", "items": { - "name": { - "type": "string" - }, - "path": { - "type": "string" + "type": "object", + "properties": { + "name": { + "description": "Name the decorator is exposed under, following '@task.'", + "type": "string" + }, + "class-name": { + "description": "Class name that implements the decorator", + "type": "string" + } } } }, @@ -444,27 +449,22 @@ "type": "array", "description": "Plugins provided by the provider", "items": { - "name": { - "type": "string", - "description": "Name of the plugin" - }, - "plugin-class": { - "type": "string", - "description": "Class to instantiate the plugin" + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Name of the plugin" + }, + "plugin-class": { + "type": "string", + "description": "Class to instantiate the plugin" + } } } }, "queues": { "type": "array", - "description": "Message Queues exposed by the provider", - "items": { - "name": { - "type": "string" - }, - "message-queue-class": { - "type": "string" - } - } + "description": "Message queue provider class names" } }, "definitions": { diff --git a/airflow-core/src/airflow/secrets/environment_variables.py b/airflow-core/src/airflow/secrets/environment_variables.py index 1a50d58138a73..6eb36978d9862 100644 --- a/airflow-core/src/airflow/secrets/environment_variables.py +++ b/airflow-core/src/airflow/secrets/environment_variables.py @@ -20,27 +20,43 @@ from __future__ import annotations import os -import re +from airflow.configuration import conf from airflow.secrets import BaseSecretsBackend CONN_ENV_PREFIX = "AIRFLOW_CONN_" VAR_ENV_PREFIX = "AIRFLOW_VAR_" +# Separates the team name from the secret id in a team namespaced environment variable +# name: AIRFLOW_CONN_____. +TEAM_SEP = "___" + class EnvironmentVariablesBackend(BaseSecretsBackend): """Retrieves Connection object and Variable from environment variable.""" + @staticmethod + def _names_a_team_namespace(secret_id: str) -> bool: + """ + Whether ``secret_id`` spells out a team scoped secret name. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped variable can exist to collide with. + """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False + return TEAM_SEP in secret_id + def get_conn_value(self, conn_id: str, team_name: str | None = None) -> str | None: + if self._names_a_team_namespace(conn_id): + return None + if team_name and ( team_var := os.environ.get(f"{CONN_ENV_PREFIX}_{team_name.upper()}___" + conn_id.upper()) ): # Format to set a team specific connection: AIRFLOW_CONN_____ return team_var - if self._names_a_team_namespace(conn_id): - return None - return os.environ.get(CONN_ENV_PREFIX + conn_id.upper()) def get_variable(self, key: str, team_name: str | None = None) -> str | None: @@ -51,39 +67,13 @@ def get_variable(self, key: str, team_name: str | None = None) -> str | None: :param team_name: Team name associated to the task trying to access the variable (if any) :return: Variable Value """ + if self._names_a_team_namespace(key): + return None + if team_name and ( team_var := os.environ.get(f"{VAR_ENV_PREFIX}_{team_name.upper()}___" + key.upper()) ): # Format to set a team specific variable: AIRFLOW_VAR_____ return team_var - if self._names_a_team_namespace(key): - return None - return os.environ.get(VAR_ENV_PREFIX + key.upper()) - - @staticmethod - def _names_a_team_namespace(secret_id: str) -> bool: - """ - Whether ``secret_id`` spells out a team namespaced environment variable name. - - A team specific secret lives in the ``____`` namespace of the - environment. An id of that shape therefore makes the team agnostic lookup -- which - prepends only ``AIRFLOW_CONN_`` / ``AIRFLOW_VAR_`` -- land inside some team's namespace, - so that lookup is refused for such an id. - - **The id is never attributed to a particular team**, because it cannot be: a team name - may itself contain the ``___`` separator, so ``_a___b___c`` is both team ``a`` with id - ``b___c`` and team ``a___b`` with id ``c``, and nothing in the string distinguishes them. - Only the caller's own namespace is ever constructed, never parsed. - - This is why the check is not "does the id belong to some team other than the caller's". - Comparing the id against the prefix the caller's team builds looks equivalent and is not: - for a caller in team ``a`` the id ``_a___b___c`` starts with ``_A___``, yet the variable - it resolves, ``AIRFLOW_CONN__A___B___C``, is team ``a___b``'s. Treating a prefix match as - ownership hands one team the secrets of every team whose name extends it. Callers reach - their own team's secrets with the bare id plus their team scope -- handled above, and safe - because that path can only ever build the caller's own namespace -- so nothing legitimate - needs a namespaced id here. - """ - return re.fullmatch(r"_.+___.+", secret_id) is not None diff --git a/airflow-core/src/airflow/serialization/serialized_objects.py b/airflow-core/src/airflow/serialization/serialized_objects.py index 54bc3389c64ce..f42e51e28f7d0 100644 --- a/airflow-core/src/airflow/serialization/serialized_objects.py +++ b/airflow-core/src/airflow/serialization/serialized_objects.py @@ -41,7 +41,7 @@ from dateutil import relativedelta from pendulum.tz.timezone import FixedTimezone, Timezone -from airflow._shared.module_loading import import_string, qualname +from airflow._shared.module_loading import qualname from airflow._shared.timezones.timezone import from_timestamp, parse_timezone, utcnow from airflow.callbacks.callback_requests import DagCallbackRequest, TaskCallbackRequest from airflow.exceptions import AirflowException, DeserializationError, SerializationError @@ -242,6 +242,31 @@ def _decode_priority_weight_strategy(var: str) -> PriorityWeightStrategy: return priority_weight_strategy_class() +# Builtin exceptions a BASE_EXC_SER node can rebuild into. A user defined subclass +# (e.g. a custom KeyError) serializes to a name absent here and won't round-trip -- +# unchanged from before, since builtins never held it either. +_DESERIALIZABLE_BUILTIN_EXCEPTIONS: dict[str, type[BaseException]] = { + "KeyError": KeyError, + "AttributeError": AttributeError, +} + + +def _resolve_airflow_exception(exc_cls_name: str) -> type[AirflowException]: + """ + Resolve a stored ``AirflowException`` name without importing it. + + Read via ``vars()`` rather than ``getattr`` -- a module's deprecation-shim + ``__getattr__`` can still import on access -- which also lets pre-3.2.0 blobs + naming the old ``airflow.exceptions`` path keep resolving. + """ + module_name, _, attr_name = exc_cls_name.rpartition(".") + module = sys.modules.get(module_name) + exc_cls = vars(module).get(attr_name) if module is not None else None + if not (isinstance(exc_cls, type) and issubclass(exc_cls, AirflowException)): + raise DeserializationError(f"Refusing to deserialize unknown exception class {exc_cls_name!r}") + return exc_cls + + def _encode_start_trigger_args(var: StartTriggerArgs) -> dict[str, Any]: """Encode a StartTriggerArgs.""" @@ -664,9 +689,14 @@ def deserialize(cls, encoded_var: Any) -> Any: kwargs = deser["kwargs"] del deser if type_ == DAT.AIRFLOW_EXC_SER: - exc_cls = import_string(exc_cls_name) + exc_cls: type[BaseException] = _resolve_airflow_exception(exc_cls_name) else: - exc_cls = import_string(f"builtins.{exc_cls_name}") + builtin_exc_cls = _DESERIALIZABLE_BUILTIN_EXCEPTIONS.get(exc_cls_name) + if builtin_exc_cls is None: + raise DeserializationError( + f"Refusing to deserialize unsupported builtin exception {exc_cls_name!r}" + ) + exc_cls = builtin_exc_cls return exc_cls(*args, **kwargs) elif type_ == DAT.SET: return {cls.deserialize(v) for v in var} diff --git a/airflow-core/src/airflow/ui/openapi-gen/queries/common.ts b/airflow-core/src/airflow/ui/openapi-gen/queries/common.ts index f11d2d808d1c1..a1d8dacf37ce7 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/queries/common.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/queries/common.ts @@ -138,13 +138,34 @@ export const UseBackfillServiceListBackfillDagRunsKeyFn = ({ backfillId, limit, export type BackfillServiceListBackfillsUiDefaultResponse = Awaited>; export type BackfillServiceListBackfillsUiQueryResult = UseQueryResult; export const useBackfillServiceListBackfillsUiKey = "BackfillServiceListBackfillsUi"; -export const UseBackfillServiceListBackfillsUiKeyFn = ({ active, dagId, limit, offset, orderBy }: { +export const UseBackfillServiceListBackfillsUiKeyFn = ({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }: { active?: boolean; + completedAtGt?: string; + completedAtGte?: string; + completedAtLt?: string; + completedAtLte?: string; + createdAtGt?: string; + createdAtGte?: string; + createdAtLt?: string; + createdAtLte?: string; dagId?: string; + fromDateGt?: string; + fromDateGte?: string; + fromDateLt?: string; + fromDateLte?: string; limit?: number; + maxActiveRunsGt?: number; + maxActiveRunsGte?: number; + maxActiveRunsLt?: number; + maxActiveRunsLte?: number; offset?: number; orderBy?: string[]; -} = {}, queryKey?: Array) => [useBackfillServiceListBackfillsUiKey, ...(queryKey ?? [{ active, dagId, limit, offset, orderBy }])]; + reprocessBehavior?: string; + toDateGt?: string; + toDateGte?: string; + toDateLt?: string; + toDateLte?: string; +} = {}, queryKey?: Array) => [useBackfillServiceListBackfillsUiKey, ...(queryKey ?? [{ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }])]; export type ConnectionServiceGetConnectionDefaultResponse = Awaited>; export type ConnectionServiceGetConnectionQueryResult = UseQueryResult; export const useConnectionServiceGetConnectionKey = "ConnectionServiceGetConnection"; diff --git a/airflow-core/src/airflow/ui/openapi-gen/queries/ensureQueryData.ts b/airflow-core/src/airflow/ui/openapi-gen/queries/ensureQueryData.ts index 637e37a62f400..b2dfef3cc4c5f 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/queries/ensureQueryData.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/queries/ensureQueryData.ts @@ -257,19 +257,61 @@ export const ensureUseBackfillServiceListBackfillDagRunsData = (queryClient: Que * @param data The data for the request. * @param data.limit * @param data.offset +* @param data.fromDateGte +* @param data.fromDateGt +* @param data.fromDateLte +* @param data.fromDateLt +* @param data.toDateGte +* @param data.toDateGt +* @param data.toDateLte +* @param data.toDateLt +* @param data.createdAtGte +* @param data.createdAtGt +* @param data.createdAtLte +* @param data.createdAtLt +* @param data.completedAtGte +* @param data.completedAtGt +* @param data.completedAtLte +* @param data.completedAtLt +* @param data.maxActiveRunsGte +* @param data.maxActiveRunsGt +* @param data.maxActiveRunsLte +* @param data.maxActiveRunsLt +* @param data.reprocessBehavior * @param data.orderBy Attributes to order by, multi criteria sort is supported. Prefix with `-` for descending order. Supported attributes: `id` * @param data.dagId * @param data.active * @returns BackfillCollectionResponse Successful Response * @throws ApiError */ -export const ensureUseBackfillServiceListBackfillsUiData = (queryClient: QueryClient, { active, dagId, limit, offset, orderBy }: { +export const ensureUseBackfillServiceListBackfillsUiData = (queryClient: QueryClient, { active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }: { active?: boolean; + completedAtGt?: string; + completedAtGte?: string; + completedAtLt?: string; + completedAtLte?: string; + createdAtGt?: string; + createdAtGte?: string; + createdAtLt?: string; + createdAtLte?: string; dagId?: string; + fromDateGt?: string; + fromDateGte?: string; + fromDateLt?: string; + fromDateLte?: string; limit?: number; + maxActiveRunsGt?: number; + maxActiveRunsGte?: number; + maxActiveRunsLt?: number; + maxActiveRunsLte?: number; offset?: number; orderBy?: string[]; -} = {}) => queryClient.ensureQueryData({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, dagId, limit, offset, orderBy }), queryFn: () => BackfillService.listBackfillsUi({ active, dagId, limit, offset, orderBy }) }); + reprocessBehavior?: string; + toDateGt?: string; + toDateGte?: string; + toDateLt?: string; + toDateLte?: string; +} = {}) => queryClient.ensureQueryData({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }), queryFn: () => BackfillService.listBackfillsUi({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }) }); /** * Get Connection * Get a connection entry. diff --git a/airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts b/airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts index d7d3fecb52c44..4ecfe6c131e3f 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/queries/prefetch.ts @@ -257,19 +257,61 @@ export const prefetchUseBackfillServiceListBackfillDagRuns = (queryClient: Query * @param data The data for the request. * @param data.limit * @param data.offset +* @param data.fromDateGte +* @param data.fromDateGt +* @param data.fromDateLte +* @param data.fromDateLt +* @param data.toDateGte +* @param data.toDateGt +* @param data.toDateLte +* @param data.toDateLt +* @param data.createdAtGte +* @param data.createdAtGt +* @param data.createdAtLte +* @param data.createdAtLt +* @param data.completedAtGte +* @param data.completedAtGt +* @param data.completedAtLte +* @param data.completedAtLt +* @param data.maxActiveRunsGte +* @param data.maxActiveRunsGt +* @param data.maxActiveRunsLte +* @param data.maxActiveRunsLt +* @param data.reprocessBehavior * @param data.orderBy Attributes to order by, multi criteria sort is supported. Prefix with `-` for descending order. Supported attributes: `id` * @param data.dagId * @param data.active * @returns BackfillCollectionResponse Successful Response * @throws ApiError */ -export const prefetchUseBackfillServiceListBackfillsUi = (queryClient: QueryClient, { active, dagId, limit, offset, orderBy }: { +export const prefetchUseBackfillServiceListBackfillsUi = (queryClient: QueryClient, { active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }: { active?: boolean; + completedAtGt?: string; + completedAtGte?: string; + completedAtLt?: string; + completedAtLte?: string; + createdAtGt?: string; + createdAtGte?: string; + createdAtLt?: string; + createdAtLte?: string; dagId?: string; + fromDateGt?: string; + fromDateGte?: string; + fromDateLt?: string; + fromDateLte?: string; limit?: number; + maxActiveRunsGt?: number; + maxActiveRunsGte?: number; + maxActiveRunsLt?: number; + maxActiveRunsLte?: number; offset?: number; orderBy?: string[]; -} = {}) => queryClient.prefetchQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, dagId, limit, offset, orderBy }), queryFn: () => BackfillService.listBackfillsUi({ active, dagId, limit, offset, orderBy }) }); + reprocessBehavior?: string; + toDateGt?: string; + toDateGte?: string; + toDateLt?: string; + toDateLte?: string; +} = {}) => queryClient.prefetchQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }), queryFn: () => BackfillService.listBackfillsUi({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }) }); /** * Get Connection * Get a connection entry. diff --git a/airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts b/airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts index 1b993afa9b01c..cd2de1aa3905d 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/queries/queries.ts @@ -257,19 +257,61 @@ export const useBackfillServiceListBackfillDagRuns = = unknown[]>({ active, dagId, limit, offset, orderBy }: { +export const useBackfillServiceListBackfillsUi = = unknown[]>({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }: { active?: boolean; + completedAtGt?: string; + completedAtGte?: string; + completedAtLt?: string; + completedAtLte?: string; + createdAtGt?: string; + createdAtGte?: string; + createdAtLt?: string; + createdAtLte?: string; dagId?: string; + fromDateGt?: string; + fromDateGte?: string; + fromDateLt?: string; + fromDateLte?: string; limit?: number; + maxActiveRunsGt?: number; + maxActiveRunsGte?: number; + maxActiveRunsLt?: number; + maxActiveRunsLte?: number; offset?: number; orderBy?: string[]; -} = {}, queryKey?: TQueryKey, options?: Omit, "queryKey" | "queryFn">) => useQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, dagId, limit, offset, orderBy }, queryKey), queryFn: () => BackfillService.listBackfillsUi({ active, dagId, limit, offset, orderBy }) as TData, ...options }); + reprocessBehavior?: string; + toDateGt?: string; + toDateGte?: string; + toDateLt?: string; + toDateLte?: string; +} = {}, queryKey?: TQueryKey, options?: Omit, "queryKey" | "queryFn">) => useQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }, queryKey), queryFn: () => BackfillService.listBackfillsUi({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }) as TData, ...options }); /** * Get Connection * Get a connection entry. diff --git a/airflow-core/src/airflow/ui/openapi-gen/queries/suspense.ts b/airflow-core/src/airflow/ui/openapi-gen/queries/suspense.ts index 4d2dc42adcaba..1e362583b417c 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/queries/suspense.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/queries/suspense.ts @@ -257,19 +257,61 @@ export const useBackfillServiceListBackfillDagRunsSuspense = = unknown[]>({ active, dagId, limit, offset, orderBy }: { +export const useBackfillServiceListBackfillsUiSuspense = = unknown[]>({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }: { active?: boolean; + completedAtGt?: string; + completedAtGte?: string; + completedAtLt?: string; + completedAtLte?: string; + createdAtGt?: string; + createdAtGte?: string; + createdAtLt?: string; + createdAtLte?: string; dagId?: string; + fromDateGt?: string; + fromDateGte?: string; + fromDateLt?: string; + fromDateLte?: string; limit?: number; + maxActiveRunsGt?: number; + maxActiveRunsGte?: number; + maxActiveRunsLt?: number; + maxActiveRunsLte?: number; offset?: number; orderBy?: string[]; -} = {}, queryKey?: TQueryKey, options?: Omit, "queryKey" | "queryFn">) => useSuspenseQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, dagId, limit, offset, orderBy }, queryKey), queryFn: () => BackfillService.listBackfillsUi({ active, dagId, limit, offset, orderBy }) as TData, ...options }); + reprocessBehavior?: string; + toDateGt?: string; + toDateGte?: string; + toDateLt?: string; + toDateLte?: string; +} = {}, queryKey?: TQueryKey, options?: Omit, "queryKey" | "queryFn">) => useSuspenseQuery({ queryKey: Common.UseBackfillServiceListBackfillsUiKeyFn({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }, queryKey), queryFn: () => BackfillService.listBackfillsUi({ active, completedAtGt, completedAtGte, completedAtLt, completedAtLte, createdAtGt, createdAtGte, createdAtLt, createdAtLte, dagId, fromDateGt, fromDateGte, fromDateLt, fromDateLte, limit, maxActiveRunsGt, maxActiveRunsGte, maxActiveRunsLt, maxActiveRunsLte, offset, orderBy, reprocessBehavior, toDateGt, toDateGte, toDateLt, toDateLte }) as TData, ...options }); /** * Get Connection * Get a connection entry. diff --git a/airflow-core/src/airflow/ui/openapi-gen/requests/services.gen.ts b/airflow-core/src/airflow/ui/openapi-gen/requests/services.gen.ts index 88ba2d33d68a8..a61b89c5799a5 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/requests/services.gen.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/requests/services.gen.ts @@ -690,6 +690,27 @@ export class BackfillService { * @param data The data for the request. * @param data.limit * @param data.offset + * @param data.fromDateGte + * @param data.fromDateGt + * @param data.fromDateLte + * @param data.fromDateLt + * @param data.toDateGte + * @param data.toDateGt + * @param data.toDateLte + * @param data.toDateLt + * @param data.createdAtGte + * @param data.createdAtGt + * @param data.createdAtLte + * @param data.createdAtLt + * @param data.completedAtGte + * @param data.completedAtGt + * @param data.completedAtLte + * @param data.completedAtLt + * @param data.maxActiveRunsGte + * @param data.maxActiveRunsGt + * @param data.maxActiveRunsLte + * @param data.maxActiveRunsLt + * @param data.reprocessBehavior * @param data.orderBy Attributes to order by, multi criteria sort is supported. Prefix with `-` for descending order. Supported attributes: `id` * @param data.dagId * @param data.active @@ -703,6 +724,27 @@ export class BackfillService { query: { limit: data.limit, offset: data.offset, + from_date_gte: data.fromDateGte, + from_date_gt: data.fromDateGt, + from_date_lte: data.fromDateLte, + from_date_lt: data.fromDateLt, + to_date_gte: data.toDateGte, + to_date_gt: data.toDateGt, + to_date_lte: data.toDateLte, + to_date_lt: data.toDateLt, + created_at_gte: data.createdAtGte, + created_at_gt: data.createdAtGt, + created_at_lte: data.createdAtLte, + created_at_lt: data.createdAtLt, + completed_at_gte: data.completedAtGte, + completed_at_gt: data.completedAtGt, + completed_at_lte: data.completedAtLte, + completed_at_lt: data.completedAtLt, + max_active_runs_gte: data.maxActiveRunsGte, + max_active_runs_gt: data.maxActiveRunsGt, + max_active_runs_lte: data.maxActiveRunsLte, + max_active_runs_lt: data.maxActiveRunsLt, + reprocess_behavior: data.reprocessBehavior, order_by: data.orderBy, dag_id: data.dagId, active: data.active @@ -4291,6 +4333,7 @@ export class XcomService { 401: 'Unauthorized', 403: 'Forbidden', 404: 'Not Found', + 409: 'Conflict', 422: 'Validation Error' } }); diff --git a/airflow-core/src/airflow/ui/openapi-gen/requests/types.gen.ts b/airflow-core/src/airflow/ui/openapi-gen/requests/types.gen.ts index c0b4e673d86fc..1dc902889c564 100644 --- a/airflow-core/src/airflow/ui/openapi-gen/requests/types.gen.ts +++ b/airflow-core/src/airflow/ui/openapi-gen/requests/types.gen.ts @@ -3102,13 +3102,34 @@ export type CreateBackfillDryRunResponse = DryRunBackfillCollectionResponse; export type ListBackfillsUiData = { active?: boolean | null; + completedAtGt?: string | null; + completedAtGte?: string | null; + completedAtLt?: string | null; + completedAtLte?: string | null; + createdAtGt?: string | null; + createdAtGte?: string | null; + createdAtLt?: string | null; + createdAtLte?: string | null; dagId?: string | null; + fromDateGt?: string | null; + fromDateGte?: string | null; + fromDateLt?: string | null; + fromDateLte?: string | null; limit?: number; + maxActiveRunsGt?: number | null; + maxActiveRunsGte?: number | null; + maxActiveRunsLt?: number | null; + maxActiveRunsLte?: number | null; offset?: number; /** * Attributes to order by, multi criteria sort is supported. Prefix with `-` for descending order. Supported attributes: `id` */ orderBy?: Array<(string)>; + reprocessBehavior?: string | null; + toDateGt?: string | null; + toDateGte?: string | null; + toDateLt?: string | null; + toDateLte?: string | null; }; export type ListBackfillsUiResponse = BackfillCollectionResponse; @@ -8082,6 +8103,10 @@ export type $OpenApiTs = { * Not Found */ 404: HTTPExceptionResponse; + /** + * Conflict + */ + 409: HTTPExceptionResponse; /** * Validation Error */ diff --git a/airflow-core/src/airflow/ui/package.json b/airflow-core/src/airflow/ui/package.json index f50df11191ee4..35393d6f81429 100644 --- a/airflow-core/src/airflow/ui/package.json +++ b/airflow-core/src/airflow/ui/package.json @@ -33,38 +33,38 @@ "@monaco-editor/react": "^4.7.0", "@tanstack/react-query": "^5.101.4", "@tanstack/react-table": "^8.21.3", - "@tanstack/react-virtual": "^3.14.8", + "@tanstack/react-virtual": "^3.14.9", "@visx/group": "^4.0.0", "@visx/shape": "^4.0.0", "@xyflow/react": "^12.11.1", "anser": "^2.3.5", - "axios": "^1.18.1", + "axios": "^1.19.0", "chakra-react-select": "^6.1.3", "chart.js": "^4.5.1", "chartjs-adapter-dayjs-4": "^1.0.4", "chartjs-plugin-annotation": "^3.1.0", "culori": "^4.0.2", "dayjs": "^1.11.21", - "elkjs": "^0.11.1", + "elkjs": "^0.12.0", "html-to-image": "^1.11.13", "i18next": "^26.3.6", "i18next-browser-languagedetector": "^8.2.1", "i18next-http-backend": "^4.0.1", - "katex": "^0.17.0", - "mermaid": "^11.16.0", - "monaco-editor": "^0.55.1", + "katex": "^0.18.1", + "mermaid": "^11.16.1", + "monaco-editor": "^0.56.0", "next-themes": "^0.4.6", "react": "^19.2.8", "react-chartjs-2": "^5.3.1", "react-dom": "^19.2.8", - "react-hook-form": "^7.83.0", + "react-hook-form": "^7.84.0", "react-hotkeys-hook": "^5.3.3", "react-i18next": "^17.0.11", "react-icons": "^5.7.0", "react-innertext": "^1.1.5", "react-markdown": "^10.1.0", "react-resizable-panels": "^3.0.6", - "react-router-dom": "^7.18.1", + "react-router-dom": "^7.18.2", "react-syntax-highlighter": "^16.1.1", "rehype-katex": "^7.0.1", "remark-gfm": "^4.0.1", @@ -78,7 +78,7 @@ "@7nohe/openapi-react-query-codegen": "^1.6.2", "@eslint/compat": "^2.1.0", "@eslint/js": "^10.0.1", - "@playwright/test": "^1.62.0", + "@playwright/test": "^1.62.1", "@rolldown/plugin-babel": "^0.2.3", "@stylistic/eslint-plugin": "^5.10.0", "@tanstack/eslint-plugin-query": "^5.101.4", @@ -86,15 +86,15 @@ "@testing-library/react": "^16.3.2", "@trivago/prettier-plugin-sort-imports": "^6.0.2", "@types/culori": "^4.0.1", - "@types/node": "^26.1.1", - "@types/react": "^19.2.17", - "@types/react-dom": "^19.2.3", + "@types/node": "^26.1.2", + "@types/react": "^19.2.18", + "@types/react-dom": "^19.2.4", "@types/react-syntax-highlighter": "^15.5.13", "@typescript-eslint/eslint-plugin": "^8.65.0", "@typescript-eslint/parser": "^8.65.0", "@typescript-eslint/utils": "^8.65.0", - "@vitejs/plugin-react": "^6.0.4", - "@vitejs/plugin-react-swc": "^4.3.2", + "@vitejs/plugin-react": "^6.0.5", + "@vitejs/plugin-react-swc": "^4.3.3", "@vitest/coverage-v8": "^4.1.9", "babel-plugin-react-compiler": "^1.0.0", "eslint": "^10.8.0", @@ -108,7 +108,7 @@ "eslint-plugin-react-hooks": "^7.1.1", "eslint-plugin-react-refresh": "^0.5.3", "eslint-plugin-unicorn": "^72.0.0", - "globals": "^17.8.0", + "globals": "^17.9.0", "happy-dom": "^20.11.1", "jsonc-eslint-parser": "^3.1.0", "msw": "^2.15.0", @@ -117,7 +117,7 @@ "ts-morph": "^27.0.2", "typescript": "^6.0.3", "typescript-eslint": "^8.65.0", - "vite": "^8.1.5", + "vite": "^8.2.0", "vite-plugin-css-injected-by-js": "^5.0.2", "vitest": "^4.1.9", "web-worker": "^1.5.0" diff --git a/airflow-core/src/airflow/ui/pnpm-lock.yaml b/airflow-core/src/airflow/ui/pnpm-lock.yaml index 7d9c3cf60a724..69072181d64ae 100644 --- a/airflow-core/src/airflow/ui/pnpm-lock.yaml +++ b/airflow-core/src/airflow/ui/pnpm-lock.yaml @@ -43,10 +43,10 @@ importers: version: 2.3.4 '@chakra-ui/react': specifier: ~3.36.1 - version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@emotion/react': specifier: ^11.14.0 - version: 11.14.0(@types/react@19.2.17)(react@19.2.8) + version: 11.14.0(@types/react@19.2.18)(react@19.2.8) '@guanmingchiu/sqlparser-ts': specifier: ^0.62.0 version: 0.62.0 @@ -55,7 +55,7 @@ importers: version: 1.2.3 '@monaco-editor/react': specifier: ^4.7.0 - version: 4.7.0(monaco-editor@0.55.1)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 4.7.0(monaco-editor@0.56.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@tanstack/react-query': specifier: ^5.101.4 version: 5.101.4(react@19.2.8) @@ -63,26 +63,26 @@ importers: specifier: ^8.21.3 version: 8.21.3(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@tanstack/react-virtual': - specifier: ^3.14.8 - version: 3.14.8(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + specifier: ^3.14.9 + version: 3.14.9(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@visx/group': specifier: ^4.0.0 - version: 4.0.0(@types/react@19.2.17)(react@19.2.8) + version: 4.0.0(@types/react@19.2.18)(react@19.2.8) '@visx/shape': specifier: ^4.0.0 - version: 4.0.0(@types/react@19.2.17)(react@19.2.8) + version: 4.0.0(@types/react@19.2.18)(react@19.2.8) '@xyflow/react': specifier: ^12.11.1 - version: 12.11.2(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 12.11.2(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) anser: specifier: ^2.3.5 version: 2.3.5 axios: - specifier: ^1.18.1 - version: 1.18.1 + specifier: ^1.19.0 + version: 1.19.0 chakra-react-select: specifier: ^6.1.3 - version: 6.1.3(@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(@types/react@19.2.17)(next-themes@0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 6.1.3(@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(@types/react@19.2.18)(next-themes@0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) chart.js: specifier: ^4.5.1 version: 4.5.1 @@ -99,8 +99,8 @@ importers: specifier: ^1.11.21 version: 1.11.21 elkjs: - specifier: ^0.11.1 - version: 0.11.1 + specifier: ^0.12.0 + version: 0.12.0 html-to-image: specifier: ^1.11.13 version: 1.11.13 @@ -114,14 +114,14 @@ importers: specifier: ^4.0.1 version: 4.0.1 katex: - specifier: ^0.17.0 - version: 0.17.0 + specifier: ^0.18.1 + version: 0.18.1 mermaid: - specifier: ^11.16.0 - version: 11.16.0 + specifier: ^11.16.1 + version: 11.16.1 monaco-editor: - specifier: ^0.55.1 - version: 0.55.1 + specifier: ^0.56.0 + version: 0.56.0 next-themes: specifier: ^0.4.6 version: 0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -135,8 +135,8 @@ importers: specifier: ^19.2.8 version: 19.2.8(react@19.2.8) react-hook-form: - specifier: ^7.83.0 - version: 7.83.0(react@19.2.8) + specifier: ^7.84.0 + version: 7.84.0(react@19.2.8) react-hotkeys-hook: specifier: ^5.3.3 version: 5.3.3(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -148,16 +148,16 @@ importers: version: 5.7.0(react@19.2.8) react-innertext: specifier: ^1.1.5 - version: 1.1.5(@types/react@19.2.17)(react@19.2.8) + version: 1.1.5(@types/react@19.2.18)(react@19.2.8) react-markdown: specifier: ^10.1.0 - version: 10.1.0(@types/react@19.2.17)(react@19.2.8) + version: 10.1.0(@types/react@19.2.18)(react@19.2.8) react-resizable-panels: specifier: ^3.0.6 version: 3.0.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) react-router-dom: - specifier: ^7.18.1 - version: 7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + specifier: ^7.18.2 + version: 7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) react-syntax-highlighter: specifier: ^16.1.1 version: 16.1.1(react@19.2.8) @@ -181,7 +181,7 @@ importers: version: 2.9.0 zustand: specifier: ^5.0.14 - version: 5.0.14(@types/react@19.2.17)(react@19.2.8)(use-sync-external-store@1.6.0(react@19.2.8)) + version: 5.0.14(@types/react@19.2.18)(react@19.2.8)(use-sync-external-store@1.6.0(react@19.2.8)) devDependencies: '@7nohe/openapi-react-query-codegen': specifier: ^1.6.2 @@ -193,11 +193,11 @@ importers: specifier: ^10.0.1 version: 10.0.1(eslint@10.8.0(jiti@1.21.7)) '@playwright/test': - specifier: ^1.62.0 - version: 1.62.0 + specifier: ^1.62.1 + version: 1.62.1 '@rolldown/plugin-babel': specifier: ^0.2.3 - version: 0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + version: 0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.2.3)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) '@stylistic/eslint-plugin': specifier: ^5.10.0 version: 5.10.0(eslint@10.8.0(jiti@1.21.7)) @@ -209,7 +209,7 @@ importers: version: 7.0.0(@testing-library/dom@10.4.0) '@testing-library/react': specifier: ^16.3.2 - version: 16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@trivago/prettier-plugin-sort-imports': specifier: ^6.0.2 version: 6.0.2(prettier@3.9.6) @@ -217,14 +217,14 @@ importers: specifier: ^4.0.1 version: 4.0.1 '@types/node': - specifier: ^26.1.1 - version: 26.1.1 + specifier: ^26.1.2 + version: 26.1.2 '@types/react': - specifier: ^19.2.17 - version: 19.2.17 + specifier: ^19.2.18 + version: 19.2.18 '@types/react-dom': - specifier: ^19.2.3 - version: 19.2.3(@types/react@19.2.17) + specifier: ^19.2.4 + version: 19.2.4(@types/react@19.2.18) '@types/react-syntax-highlighter': specifier: ^15.5.13 version: 15.5.13 @@ -238,11 +238,11 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@10.8.0(jiti@1.21.7))(typescript@6.0.3) '@vitejs/plugin-react': - specifier: ^6.0.4 - version: 6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + specifier: ^6.0.5 + version: 6.0.5(@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.2.3)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) '@vitejs/plugin-react-swc': - specifier: ^4.3.2 - version: 4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + specifier: ^4.3.3 + version: 4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) '@vitest/coverage-v8': specifier: ^4.1.9 version: 4.1.10(vitest@4.1.10) @@ -283,8 +283,8 @@ importers: specifier: ^72.0.0 version: 72.0.0(eslint@10.8.0(jiti@1.21.7)) globals: - specifier: ^17.8.0 - version: 17.8.0 + specifier: ^17.9.0 + version: 17.9.0 happy-dom: specifier: ^20.11.1 version: 20.11.1 @@ -293,7 +293,7 @@ importers: version: 3.1.0 msw: specifier: ^2.15.0 - version: 2.15.0(@types/node@26.1.1)(typescript@6.0.3) + version: 2.15.0(@types/node@26.1.2)(typescript@6.0.3) openapi-merge-cli: specifier: ^1.3.2 version: 1.3.2 @@ -310,14 +310,14 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@10.8.0(jiti@1.21.7))(typescript@6.0.3) vite: - specifier: ^8.1.5 - version: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + specifier: ^8.2.0 + version: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) vite-plugin-css-injected-by-js: specifier: ^5.0.2 - version: 5.0.2(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + version: 5.0.2(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) vitest: specifier: ^4.1.9 - version: 4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3))(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + version: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3))(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) web-worker: specifier: ^1.5.0 version: 1.5.0 @@ -426,6 +426,11 @@ packages: engines: {node: '>=6.0.0'} hasBin: true + '@babel/parser@7.29.8': + resolution: {integrity: sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA==} + engines: {node: '>=6.0.0'} + hasBin: true + '@babel/runtime@7.26.10': resolution: {integrity: sha512-2WJMeRQPHKSPemqk/awGrAiuFfzBmOIPXKizAsVhWH9YJqLZ0H+HS4c8loHGgW6utJ3E/ejXQUsiGaQy2NZ9Fw==} engines: {node: '>=6.9.0'} @@ -462,6 +467,10 @@ packages: resolution: {integrity: sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA==} engines: {node: '>=6.9.0'} + '@babel/types@7.29.8': + resolution: {integrity: sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg==} + engines: {node: '>=6.9.0'} + '@bcoe/v8-coverage@1.0.2': resolution: {integrity: sha512-6zABk/ECA/QYSCQ1NGiVwwbQerUCZ+TQbp64Q3AgmfNvurHH0j8TtXa1qbShXA6qqkpAj4V5W8pP6mLe1mcMqA==} engines: {node: '>=18'} @@ -482,15 +491,6 @@ packages: '@chevrotain/types@11.1.2': resolution: {integrity: sha512-U+HFai5+zmJCkK86QsaJtoITlboZHBqrVketcO2ROv865xfCMSFpELQoz1GkX5GzME8pTa+3kbKrZHQtI0gdbw==} - '@emnapi/core@1.11.1': - resolution: {integrity: sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==} - - '@emnapi/runtime@1.11.1': - resolution: {integrity: sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==} - - '@emnapi/wasi-threads@1.2.2': - resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} - '@emotion/babel-plugin@11.13.5': resolution: {integrity: sha512-pxHCpT2ex+0q+HH91/zsdHkw/lXd468DIN2zvfvLtPKLLMo6gQj7oLObq8PhkrxOZb/gGCq03S3Z7PDhS8pduQ==} @@ -645,8 +645,8 @@ packages: '@iconify/types@2.0.0': resolution: {integrity: sha512-+wluvCrRhXrhyOmRDJ3q8mux9JkKy5SJ/v8ol2tu4FVjyYvtEzkc/3pK15ET6RKg4b4w4BmTk1+gsCUhf21Ykg==} - '@iconify/utils@3.1.3': - resolution: {integrity: sha512-LPKOXPn/zV+zis1oOfGWogaXVpqUybF3ZS6SCZIsz8vg0ivVp9+fVqyYB7xq0aiST/VhUQYGO1qo6uoYSiEJqw==} + '@iconify/utils@3.1.4': + resolution: {integrity: sha512-b1S7B1k9ohZ+iNTi2ATxbRYG9fTrJmUT0rc46bvVnNxqNRGW7dyo/vRREwyniI5IRN2RSJHDcm+s3BjWrSAjHw==} '@inquirer/ansi@2.0.7': resolution: {integrity: sha512-3eTuUO1vH2cZm2ZKHeQxnOqlTi9EfZDGgIe3BL3I4u+rJHocr9Fz86M4fjYABPvFnQG/gGK551HqDiIcETwU6Q==} @@ -742,12 +742,6 @@ packages: resolution: {integrity: sha512-VVPPgHyQ6ShqnrmDWuxjmUIsO9gWyOZFmuOfLd9LfBGQJwZfy0gvv9pbHSJuoFNIYC7ZDX9aoFwowjcdSC4E8w==} engines: {node: '>=18'} - '@napi-rs/wasm-runtime@1.1.6': - resolution: {integrity: sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==} - peerDependencies: - '@emnapi/core': ^1.7.1 - '@emnapi/runtime': ^1.7.1 - '@open-draft/deferred-promise@2.2.0': resolution: {integrity: sha512-CecwLWx3rhxVQF6V4bAgPS5t+So2sTbPgAzafKkVizyi7tlwpcFpdFqq+wqF2OwNBmqFuu6tOyouTuxgpMfzmA==} @@ -764,8 +758,8 @@ packages: resolution: {integrity: sha512-XRO0zi2NIUKq2lUk3T1ecFSld1fMWRKE6naRFGkgkdeosx7IslyUKNv5Dcb5PJTja9tHJoFu0v/7yEpAkrkrTg==} engines: {node: ^20.19.0 || ^22.13.0 || >=24} - '@oxc-project/types@0.139.0': - resolution: {integrity: sha512-r9gHphtCs+1M7J0pw6Sn/hh/Wpa/iQrOOkrNAlVLF/gHq+/CJmHIWKKUUhdWjcD6CIa8idarspCsASiXCXvFUw==} + '@oxc-project/types@0.143.0': + resolution: {integrity: sha512-u6JZdLBTLotrNC9Vd6vPssINdzcCzleKAH6EJKImQb7GtYvX5keN2dxkoK44stCc4tffE6QQRtZTXVSzsLUlWA==} '@pandacss/is-valid-prop@1.11.5': resolution: {integrity: sha512-8m1/gG0wSRMa1QypCkWzeKUeMKY7k14nxsJCHs7Z1g1fI9I7WsLjW/VI58bLZdJxAxtoreMtm6TfIITda8P4zQ==} @@ -775,96 +769,97 @@ packages: resolution: {integrity: sha512-SEeaJLb3qBNF/OaXnaR1NmmBbFYk1zC0ZH/52fATcRPLFg/p791YrcyFFy44Bo9sLaGuSuLp5Q6axbb/O+v/RA==} engines: {node: ^14.18.0 || >=16.0.0} - '@playwright/test@1.62.0': - resolution: {integrity: sha512-9zOJ6ZQRAena31MpOH9VSzIz8Ou3YJ/wtY/eQm5T2uhfhG7/U3COrMS8xOtUrZrp9OgdmzEnIYODye3nY1VqzA==} + '@playwright/test@1.62.1': + resolution: {integrity: sha512-DTcUc8qii+cpHvtOwggMtBRMjKZHXYWdw8syRYu2vtzuq4Wxphqq4NfCs5Zt44L6mA8rfDfj+PHnxFc/FeK6mQ==} engines: {node: '>=20'} hasBin: true - '@rolldown/binding-android-arm64@1.1.5': - resolution: {integrity: sha512-lZg8fqIv2v7FF237bwMgzGZEJvGL79/s5knJ/i6FmsGF4XXlzccZ4jb+TrFIxtSSxFtIpdsgrPZeMk1I9AFcyQ==} + '@rolldown/binding-android-arm64@1.2.3': + resolution: {integrity: sha512-zrJtHDcaZJ1Fp7xf4hNl+7seH9Cn/N5TwLYkhgXREtBwAd/jaqW3uqeHxpDugJLVICWg4eW44kOQEGJ1r6jCGw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@rolldown/binding-darwin-arm64@1.1.5': - resolution: {integrity: sha512-51Bnx9pNiMRKSUNtBfySkNJ9vMU9Hh3I1ozDd6gyPPYzaXCfnptUcEZxXGYFn+ul2dtcMUiqGR1Yai2K10uoTw==} + '@rolldown/binding-darwin-arm64@1.2.3': + resolution: {integrity: sha512-ieIiibVCp0tX7TLu2cafoNPv8wJyYi01ekXpbf8q2j7F4rGAhhXb/eQh7ge9DRBY78GwmRQtvjZDux7EDbA8kA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@rolldown/binding-darwin-x64@1.1.5': - resolution: {integrity: sha512-Tm+gbfC0aHu1tBA/JvKQh32S0K6YgCHkiAF4/W6xX0K0RmNuc94VeK419dJoE65R5aRxmo+noZQSWrAMF6yb6g==} + '@rolldown/binding-darwin-x64@1.2.3': + resolution: {integrity: sha512-Zh9tCon19eDXJoihx0rqKhMUlMYqzwj3aPsSuHmI4RWZh62dWUL+DJN4C5YQya5TcQBJU/Fe8+rY0jhXTQITqA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@rolldown/binding-freebsd-x64@1.1.5': - resolution: {integrity: sha512-JMzDKCCXq93YccG5gz3hvOs1oXRKAf0XYpfOS88e+wZrC8Iugj6j68867vrYZkvpDDpKn/KoKORThmchMpF6TA==} + '@rolldown/binding-freebsd-x64@1.2.3': + resolution: {integrity: sha512-nGbJWewA1wrXXZiQhjAT5rhibGfns5ZNkDVqxsO6zJ3f3YvpoDNNmGMSbbhLuXKjNScaBJVOAboztAWVespQMg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': - resolution: {integrity: sha512-uML21j2K5TfPGutKxub+M+nLjZIrWjXQ5Grx4lCe/nimTj9B4L63zHpjXLl4y0L3mcm2htEQIb06oCG/szerNw==} + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': + resolution: {integrity: sha512-QNniJr5Kml0kDEB98jiDOJjXNroxIIi0IXIbdYzY26Xt1pVbeP62+KnoIZLwirOymX/0jDk/2gI/bNUv7A7OIw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@rolldown/binding-linux-arm64-gnu@1.1.5': - resolution: {integrity: sha512-navSiuTMogvnQoZoM/v+l3ZWo50/NTwSHSzheABx/RCnmUPaKwq9qSo4Br2OYRs21+Fz8uFqITZM3H4opOB0/Q==} + '@rolldown/binding-linux-arm64-gnu@1.2.3': + resolution: {integrity: sha512-TkqEAcmmvH3I/q4114NB4RVt6241Dao48pF45uLcFGrwAaIn0iITgTAKP/dLjbN0R4buJjGb91+UHSoFmpgIWw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] + libc: [glibc] - '@rolldown/binding-linux-arm64-musl@1.1.5': - resolution: {integrity: sha512-lAryqH7IteztmCXQXk0etKj4wBQ7Gx5S6LjKhsgp9zb8I5bsuvU/2llH1hDQcjsFeqIsovMVN339/8pUDDBXxA==} + '@rolldown/binding-linux-arm64-musl@1.2.3': + resolution: {integrity: sha512-NHqjnxpsndf4MPymxteFAWHHfkTL8HjWh1KB7z23ofZ6QO2euONuxDXjat69dKZRALnGypg8k8SsK8vZJoXv1Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] + libc: [musl] - '@rolldown/binding-linux-ppc64-gnu@1.1.5': - resolution: {integrity: sha512-fsK/sNBnxzBlL4O1JNrZakVQxPspqpED5dLtNsZS9oOKmtSpdNIzxH2kkol5HYTWJN47sE20ztMJPxfZ89qGOg==} + '@rolldown/binding-linux-ppc64-gnu@1.2.3': + resolution: {integrity: sha512-6tbrbwfz5GB9DQ4Jwo6hy9v+vR31xZlvzZ6n5Xut6Hhx5PvrA9q/HsK8KMaYQp063iqZGXwNvZtYNLD7EM/x0w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] + libc: [glibc] - '@rolldown/binding-linux-s390x-gnu@1.1.5': - resolution: {integrity: sha512-gLYb4BIadlfTOYT5gO503n8zQjXflgzpD0FcyKh0Mzx3rqCZKnHoJWV9xe1KXUJ5lx2JfcSHr/mhzS0PC/McAA==} + '@rolldown/binding-linux-s390x-gnu@1.2.3': + resolution: {integrity: sha512-oyuXxXmoZHjXC917IAPFAAv4wWAa0cM9afk8nx1+9/jNNOX1uPf8yDA6p7G0RypOfw/X0PQt5IfoquY1um+zSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] + libc: [glibc] - '@rolldown/binding-linux-x64-gnu@1.1.5': - resolution: {integrity: sha512-FjcpEKUyJygHgs1o50VYNvkt5+7Le/VEdYt0AkRpkL33MnyQfwr8l5mXwMmfmTbyMPr5vJLC+8/Gd9gXnwU1QQ==} + '@rolldown/binding-linux-x64-gnu@1.2.3': + resolution: {integrity: sha512-TytMwF2KVGqP2tgd0I1OY0PAv78dZRAYcF5ssDzjM34SUXCED3uXvSd5+lHoC0bTD6eEdFz7LdQNCO1y0oVk9w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] + libc: [glibc] - '@rolldown/binding-linux-x64-musl@1.1.5': - resolution: {integrity: sha512-Me+PfPI2TMeOQk0gYWfLQZtTktrmzbr8cDboqX83XKc7UrgAi55gF+2dUkWdxd19n55Essp2yeca+O9N5rBxHg==} + '@rolldown/binding-linux-x64-musl@1.2.3': + resolution: {integrity: sha512-/E9m3qstrJFVPoULV25mVQblSNExY2+kBsYe4sy0Tn0yOOgJ8wZbZt3KnRbF/XeU2Gl1STKUQnDNTqhIE5MD4A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] + libc: [musl] - '@rolldown/binding-openharmony-arm64@1.1.5': - resolution: {integrity: sha512-yc5WrLzXks6zCQfn9Oxr8pORKyl/pF+QjHmW/Qx3qu0oyrrNC+y2JLTU1E2rcWYAmzlnqngWXHQjy51VzW70Vw==} + '@rolldown/binding-openharmony-arm64@1.2.3': + resolution: {integrity: sha512-Kr0OcsoQI816i6HOl3vFHpd1K0eZyh76zgfj4c1nTyaTsd5r2Mj1lwM4R90y/qaCfmTn9eHy0SKwi98eitRxug==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@rolldown/binding-wasm32-wasi@1.1.5': - resolution: {integrity: sha512-VbQGPX2b4r48TAMIM2cjgluIM1HYutm4pcTEJsle7iEP7sB1dFqtPLBVbdLAZCxy1txCcPxf4QFf4v8uvltPqA==} - engines: {node: ^20.19.0 || >=22.12.0} - cpu: [wasm32] - - '@rolldown/binding-win32-arm64-msvc@1.1.5': - resolution: {integrity: sha512-gHv82k63z4qpV5+Q1y/12KrK0ltWBukVDI8nZcbT7Tt/ZlOIVwppazneq0F93oDxTo3IgAMEDIoQh3E2n6mVsw==} + '@rolldown/binding-win32-arm64-msvc@1.2.3': + resolution: {integrity: sha512-hOtMwTqnME+/gJcH/PCZ0wn0zPUjiWOgkHpxbSJpfGKMezHltx1S7/k1SitzVa7Ww2cqrDDaFbZEhcJZO8o+Jw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@rolldown/binding-win32-x64-msvc@1.1.5': - resolution: {integrity: sha512-tTZuDBPw85tEN5PQi1pnEBzDy0Z49HtScLAbD5t6hyeU92A95pRWaSMw1GZZi/RwgSgUIl0xrSlXIT/9QzvYSA==} + '@rolldown/binding-win32-x64-msvc@1.2.3': + resolution: {integrity: sha512-ekcqMMkI2PlhYnfzQnB/cEdYUVVJViWvoUyLrbzgDoi3Snfc1mVBwdnc306ufA5ejy8JSPjT2RlW1nQSjW7efg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -898,80 +893,86 @@ packages: peerDependencies: eslint: ^9.0.0 || ^10.0.0 - '@swc/core-darwin-arm64@1.15.46': - resolution: {integrity: sha512-IsISIT22EfktVJrlvIpnAxG2u/A9aob9l99HMlx80x72WlFmFPk1V3UhkEzx86eJP8hw049KTFv/RISho2cq2Q==} + '@swc/core-darwin-arm64@1.15.47': + resolution: {integrity: sha512-GsoMtan3ojGGMGFbl31mmRu5ctZ56re8grGE8mO/OHJ8O+JRkzod02fe7X6ZQ8JvamA3imkEkx/h3u+vsOgPgA==} engines: {node: '>=10'} cpu: [arm64] os: [darwin] - '@swc/core-darwin-x64@1.15.46': - resolution: {integrity: sha512-4Tj4ppVIPCmUMpmGFiGtyEriwLyJ+yi/US4WfBrP/ok8COGddDZXLEzQETnKyK46mjvr1v0jevrS23zjoff7vA==} + '@swc/core-darwin-x64@1.15.47': + resolution: {integrity: sha512-leTi7Rx3KF4zcC637iqWgk9SoV8VXAD8ppQYXsep63px5A/UftOcxLN1pmr8Z1si/YvX90ompP/rHgpYkgwXWg==} engines: {node: '>=10'} cpu: [x64] os: [darwin] - '@swc/core-linux-arm-gnueabihf@1.15.46': - resolution: {integrity: sha512-i8tUGnNjyOgMmfmgFSg4aeJLQoFyfpIHK5FjpQAwpRyQIqEUB2w1e8zIDQzY1WhOxx8NoS1S5iUL813Un4Sf5A==} + '@swc/core-linux-arm-gnueabihf@1.15.47': + resolution: {integrity: sha512-hBqHuoWKKIsKmDBn9qVeWqj5GWZhtlcczVaqQmNRXsDfq+voR5CxKRfamA367QjJXtceYuliLFfEL8QsskRM2g==} engines: {node: '>=10'} cpu: [arm] os: [linux] - '@swc/core-linux-arm64-gnu@1.15.46': - resolution: {integrity: sha512-c0OnhqzdhfOvv6qhNCcByepB+sNYOGZyhtr2Qa6ZCHvAWTYhSRw4j/u92Stue9PbZ/6q74b9nHzi76+kVzqQHQ==} + '@swc/core-linux-arm64-gnu@1.15.47': + resolution: {integrity: sha512-TBxvRz+B4K205TWHHZxWVxkC2RFNP/Mz3PNcECBos5PsKwxjg3QSJzdoebr0VCf0Bfh8HOPldKxAP/8XkFe9gA==} engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [glibc] - '@swc/core-linux-arm64-musl@1.15.46': - resolution: {integrity: sha512-imyRpNEcUzFQFV2LE4jL68ErvmKEuZCbvZru77iQREunJ+bR4i658cupTgtG1mLYM3F1Tzy3Sb9xYb02KghWTg==} + '@swc/core-linux-arm64-musl@1.15.47': + resolution: {integrity: sha512-3Yu3Uq/VgytqsPjTMbkPU1ExADytbdWbruJYhA584E9jrpE2Ki+R6VVPoZCeAVk1Cb7QxcRTgblw6bSa6a/R+w==} engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [musl] - '@swc/core-linux-ppc64-gnu@1.15.46': - resolution: {integrity: sha512-ctEfcl/HcUeomK33cbySiHZm98GEDIxTm1EkpBsYCiHxElYBzvTXVeuQT2YwbUXn9XCrjiw4ipyUNk33k26qRg==} + '@swc/core-linux-ppc64-gnu@1.15.47': + resolution: {integrity: sha512-wfdMi5IaOaNtmh2/6geRoxIdNfqylUZFdtzTKS655y1axWfIWyx7As74vv0wVdjeCIZ3WmCI9odDd4rUttXOSQ==} engines: {node: '>=10'} cpu: [ppc64] os: [linux] + libc: [glibc] - '@swc/core-linux-s390x-gnu@1.15.46': - resolution: {integrity: sha512-DxlMdnt84TtRVTv7WL/thWyz9+QU8QZNNoAP9rrk0P68LziuhfePp8MjQ44zIprpTHTsEwyziIuGUUN5iSC1bQ==} + '@swc/core-linux-s390x-gnu@1.15.47': + resolution: {integrity: sha512-3hHYBY0yx8Ez7GMRrkhXHQzMdR5IZA6Wq5Ee4svlgwvSECLpnAJ9+0AimEGUFDvuLwE7nV/2+PYe8+Nm4rvNcQ==} engines: {node: '>=10'} cpu: [s390x] os: [linux] + libc: [glibc] - '@swc/core-linux-x64-gnu@1.15.46': - resolution: {integrity: sha512-SKxI7J6t90XPl8hRUqtJi9NfGdunN/E/vZMc7Bc0figeRdOPDBT+Tm8g7cx9xM0T0mewh2l+8dewa3Am27/P+A==} + '@swc/core-linux-x64-gnu@1.15.47': + resolution: {integrity: sha512-TjfhjgP/jGCfFHYC3JQPhJA1HwErbIJ9JfREDc1KNkvY6P0LodCgKVIlQ5deeTbkG7ih3bF5PHJLuLpaZjdRyQ==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [glibc] - '@swc/core-linux-x64-musl@1.15.46': - resolution: {integrity: sha512-qj9T6B7bosI0VEsrWOVXZN1OXxS8Tp63ywyrLxNdOycnUtLdkgYcoBsN5y8ImnDDsnwrEWZOy1e+J4xSe7mA3Q==} + '@swc/core-linux-x64-musl@1.15.47': + resolution: {integrity: sha512-CQpS8Ge/avfjZd0UEwG/sds83Uu32deQXcV1Jo3jD0mmvQQqtYAjpsDZXugmheeAwmt+YIuoVtVHro8LMYHqsQ==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [musl] - '@swc/core-win32-arm64-msvc@1.15.46': - resolution: {integrity: sha512-8p7l4c3LU+eA5g9Et1JPhNeMC1oQwXTGU+uah8DPIBX7YXzqswvaBtyKVmXefVGi/DJU1x3YJsc3mbAp9aWzSQ==} + '@swc/core-win32-arm64-msvc@1.15.47': + resolution: {integrity: sha512-0W8IKHsUTYiT7G2RqtOoVWk+89yzZikIiDUb/sCK6BmQDBhN91hQSfyUtW12jhEWLzYgcfmisfsZrmZE+84U1A==} engines: {node: '>=10'} cpu: [arm64] os: [win32] - '@swc/core-win32-ia32-msvc@1.15.46': - resolution: {integrity: sha512-tUEnfr3Bn9u6FOjUb3PN9p+09qZC2j+wNDLKHzXXZn22rqGcUqR/ohCRSS+nG9B9+X+U+3FewNEHJkTmdIvMjQ==} + '@swc/core-win32-ia32-msvc@1.15.47': + resolution: {integrity: sha512-ZIp49d2Z4/ka2jO9otOg4hDvTdPmp86kVOgS2M5FCPI7eKKZ1W0boxWn+8XeZrfERtFGW0AlMRm4JhlJa7l3NA==} engines: {node: '>=10'} cpu: [ia32] os: [win32] - '@swc/core-win32-x64-msvc@1.15.46': - resolution: {integrity: sha512-Vux7UDzBJYQggSuPfcl2w9iu+IJpgpRCxHzgCaVkELnAXAE4XZMOTX9HNcaNiwfeIDqdu2rkr69RuDm6wY8neA==} + '@swc/core-win32-x64-msvc@1.15.47': + resolution: {integrity: sha512-2h8Iek95vnixkBRCo+H8p09+Q5ll2NgSMFrWTy0iKt7+/t+8/T5mBpiT6c0ZxSS7wcWjwZ9sGZkK70tTSYHdDw==} engines: {node: '>=10'} cpu: [x64] os: [win32] - '@swc/core@1.15.46': - resolution: {integrity: sha512-Ri3em2mBpq3h2zSPliCYl63otDGqek8PPEfv2nWgRQEbZ/VBCNyypVTVQ6cEbTCXBhy+WE2T3fQb08moIyuYaw==} + '@swc/core@1.15.47': + resolution: {integrity: sha512-FbsO5JcfOjfH38W/rohBRBweJeERsAuIP4f377lmkmxTcq9exjtx4SkRuZY5CdfhR2CBVwDIJegBpJDffwNsOg==} engines: {node: '>=10'} peerDependencies: '@swc/helpers': '>=0.5.17' @@ -985,8 +986,8 @@ packages: '@swc/helpers@0.5.23': resolution: {integrity: sha512-5lSsMOTXURePglDfvuAQUqkGek9Hg2kksOYay2m0+XR++b2NWYL/4sWyuvVBIs8oKnJaxkdi9whaL/sqN13afw==} - '@swc/types@0.1.27': - resolution: {integrity: sha512-K6h3iUlqeM946U4sXFYeahefR1YBbXJvko+hv8WS8/0BNJ4OHiHRywMnQUJCqkR7Y9+hqQ1TvEpiKqUhz7NEFg==} + '@swc/types@0.1.28': + resolution: {integrity: sha512-V6Mnml8v09QALx6K0elJ7o9K/MkVDtW3t6L+7Ou/JcWtb3xwId2AH4FeOceySd2JaO87IMw4+6vSZxLm34LPbw==} '@tanstack/eslint-plugin-query@5.101.4': resolution: {integrity: sha512-jqAZJWXGd5PthJYH9wmC2O5Ry5xAN2/7zxi9qcANQ+Xix8V5JkqNRjZ8Fh+rYzqzVYGiHqbrHekt+uh38OZVpw==} @@ -1012,8 +1013,8 @@ packages: react: '>=16.8' react-dom: '>=16.8' - '@tanstack/react-virtual@3.14.8': - resolution: {integrity: sha512-O39GJQpAYEJcIu3uN1//YtmhjSEOyw75vg9CKCatBDPiD5hKtZQoJHfferyrB/LdOD3UWaoMLWtdEjarwIwdDw==} + '@tanstack/react-virtual@3.14.9': + resolution: {integrity: sha512-qZyr0FZDP8rDC4WBhsryIZmAd9bveJvFGUJJtskWaew6/0dTRS6wZxnR6VQ5bY2KwL3LjerrHqQLk3a0GKcPXQ==} peerDependencies: react: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 react-dom: ^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 @@ -1022,8 +1023,8 @@ packages: resolution: {integrity: sha512-ldZXEhOBb8Is7xLs01fR3YEc3DERiz5silj8tnGkFZytt1abEvl/GhUmCE0PMLaMPTa3Jk4HbKmRlHmu+gCftg==} engines: {node: '>=12'} - '@tanstack/virtual-core@3.17.6': - resolution: {integrity: sha512-h0/Ebo18CkOrChlQIhNtQkM5ySUnh/GumQ/D1st3hG2HWUPEF+ILUc2k29UtivCi/9G7w7G3/f7Xyd5cCFbKBw==} + '@tanstack/virtual-core@3.17.7': + resolution: {integrity: sha512-bp+v10y65sp2H7WpWfIMyxTNfl8ZVfxFTLRjPIFRryi6FV/J33z4IS53WO4pTk36KlvJ4iLiQz+oaydDC1xbcA==} '@testing-library/dom@10.4.0': resolution: {integrity: sha512-pemlzrSESWbdAloYml3bAJMEfNh1Z7EduzqPKprCH5S341frlpYnUEW0H72dLxa6IsYr+mPno20GiSm+h9dEdQ==} @@ -1072,9 +1073,6 @@ packages: '@ts-morph/common@0.28.1': resolution: {integrity: sha512-W74iWf7ILp1ZKNYXY5qbddNaml7e9Sedv5lvU1V8lftlitkc9Pq1A+jlH23ltDgWYeZFFEqGCD1Ies9hqu3O+g==} - '@tybys/wasm-util@0.10.3': - resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} - '@types/aria-query@5.0.4': resolution: {integrity: sha512-rfT93uj5s0PRL7EzccGMs3brplhcrghnDoV26NqKhCAS1hVo+WdNsPvE/yb6ilfr5hi2MEk6d5EWJTKdxg8jVw==} @@ -1141,6 +1139,9 @@ packages: '@types/d3-geo@3.1.0': resolution: {integrity: sha512-856sckF0oP/diXtS4jNsiQw/UuK5fQG8l/a9VVLeSouf1/PPbBE1i1W852zVwKwYCBkFJJB7nCFTbk6UMEXBOQ==} + '@types/d3-geo@3.1.1': + resolution: {integrity: sha512-65Emv9fQiQQqphLlRkuQ5ypPsOmWPhtBGCMv61JDPEPMvsx+gzhGf74yw1a78xFKPj6zw4AgQICJoQv0vK9M2w==} + '@types/d3-hierarchy@3.1.7': resolution: {integrity: sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg==} @@ -1159,8 +1160,8 @@ packages: '@types/d3-quadtree@3.0.6': resolution: {integrity: sha512-oUzyO1/Zm6rsxKRHA1vH0NEDG58HrT5icx/azi9MF1TWdtttWl0UIUsjEQBBh+SIkrpd21ZjEv7ptxWys1ncsg==} - '@types/d3-random@3.0.3': - resolution: {integrity: sha512-Imagg1vJ3y76Y2ea0871wpabqp613+8/r0mCLEBfdtqC7xMSfj9idOnmBYyMoULfHePJyxMAw3nWhJxzc+LFwQ==} + '@types/d3-random@3.0.4': + resolution: {integrity: sha512-UHYId5WTCx4L4YNel7NU00XUXXgvgpgZOvp10PuvsQENjMDXhh2RyFc0KBjO7B45ne4Ha1yVH7ii0vnzKkuzWA==} '@types/d3-scale-chromatic@3.1.0': resolution: {integrity: sha512-iWMJgwkK7yTRmWqRB5plb1kadXyQ5Sj8V/zYlFGMUBbIPKQScw+Dku9cAAMgJG+z5GYDoMjWGLVOvjghDEFnKQ==} @@ -1237,8 +1238,8 @@ packages: '@types/ms@2.1.0': resolution: {integrity: sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA==} - '@types/node@26.1.1': - resolution: {integrity: sha512-nxAkRSVkN1Y0JC1W8ky/fTfkGsMmcrRsbx+3XoZE+rMOX71kLYTV7fLXpqud1GpbpP5TuffXFqfX7fH2GgZREw==} + '@types/node@26.1.2': + resolution: {integrity: sha512-Vu4a5UFA9rIIFJ7rB/Vaafh9lrCQszopTCx6KjFboXTGQbPNasehVR5TEiithSDGyd1DEiUByggTZsg8jukeIg==} '@types/parse-json@4.0.2': resolution: {integrity: sha512-dISoDXWWQwUquiKsyZ4Ng+HX2KsPL7LyHKHQwgGFEA3IaKac4Obd+h2a/a6waisAoepJlBcx9paWqjA8/HVjCw==} @@ -1246,8 +1247,8 @@ packages: '@types/prismjs@1.26.6': resolution: {integrity: sha512-vqlvI7qlMvcCBbVe0AKAb4f97//Hy0EBTaiW8AalRnG/xAN5zOiWWyrNqNXeq8+KAuvRewjCVY1+IPxk4RdNYw==} - '@types/react-dom@19.2.3': - resolution: {integrity: sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==} + '@types/react-dom@19.2.4': + resolution: {integrity: sha512-Bsc+QHgp+P/F02XDzNCY9jnZNCUuLki36KT7VKrTXXLdHf+vHMNZnW1rVu5DNW/rCK+fya3DATySbLM4yhtKUw==} peerDependencies: '@types/react': ^19.2.0 @@ -1259,8 +1260,8 @@ packages: peerDependencies: '@types/react': '*' - '@types/react@19.2.17': - resolution: {integrity: sha512-MXfmqaVPEVgkBT/aY0aGCkRWWtByiYQXo3xdQ8r5RzuFrPiRn8Gar2tQdXSUQ2GKV3bkXckek89V8wQBY2Q/Aw==} + '@types/react@19.2.18': + resolution: {integrity: sha512-AnzbBERsrLKtk2XSfTbYRLjQPdy116Sty4q+T+Bp3IC4l6jNBvreVPAHmpq9qhXQM7CXZPjLVmGMw9sy+hxQ3w==} '@types/set-cookie-parser@2.4.10': resolution: {integrity: sha512-GGmQVGpQWUe5qglJozEjZV/5dyxbOOZ0LHe/lqyWssB88Y4svNfst0uqBVscdDeIKl5Jy5+aPSvy7mI9tYRguw==} @@ -1380,14 +1381,14 @@ packages: '@visx/vendor@4.0.0': resolution: {integrity: sha512-LoBNzWjTXzBfu095BQxB14IwZQHHOLs8ZMzM6t2FL4ZGORaACgLcmXRRLhFo0VmRR8L7iNTM4HHc1j6yFvzsAA==} - '@vitejs/plugin-react-swc@4.3.2': - resolution: {integrity: sha512-MlSlmSAYbYwPjGa+CjOAqwij09iPYAQDHwx8osVkwoDamCxXlg+avpkC65Ysv+L/uqUWvRTsnJCKPjVhZih/sA==} + '@vitejs/plugin-react-swc@4.3.3': + resolution: {integrity: sha512-bti8ZAcvz4Lh6/e4Uk2k3aa1TiUXbbMsahuqOHvd3MveFTkKDZOA6wQVkpj7J/+tepX/wGfe+lsGh/t24HTXMQ==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: vite: ^4 || ^5 || ^6 || ^7 || ^8 - '@vitejs/plugin-react@6.0.4': - resolution: {integrity: sha512-XcCQz0TBpBgljhj0gMuuDj49i6Ytqh5q1osT/Gp5uAVJUCTWxyskk/l1jwYYiu2xcNHHipdMz40EGfM1VdamVg==} + '@vitejs/plugin-react@6.0.5': + resolution: {integrity: sha512-BOVzne/NL162sMdResB25mUv+vWMF5NoAjNf09TeGlE7ZpszZWSD3winycicLJw72yeVsoCn/2kOhEuCvEShMA==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: '@rolldown/plugin-babel': ^0.1.7 || ^0.2.0 @@ -1792,8 +1793,8 @@ packages: resolution: {integrity: sha512-Xm7bpRXnDSX2YE2YFfBk2FnF0ep6tmG7xPh8iHee8MIcrgq762Nkce856dYtJYLkuIoYZvGfTs/PbZhideTcEg==} engines: {node: '>=4'} - axios@1.18.1: - resolution: {integrity: sha512-3nTvFlvpn9Zu/RkHUqtc7/+al4UpRW5az71ap5zccp6e8RAYEzhMTecX8Dz1wWDYrPpUoB1HAQEGEAEvUr7S9g==} + axios@1.19.0: + resolution: {integrity: sha512-ht/iuYZXEjFxLH/Hkezgd7m6JKlHHXEUSneaDz8uZe1Gj5QZtCnpyDsckvAiEnT89OEbCLmnte4R4sn7P0EKFw==} axobject-query@4.1.0: resolution: {integrity: sha512-qIj0G9wZbMGNLjLmg1PT6v2mE9AH2zlnADJD/2tC6E00hgmhUOfEB6greHPAfLRSufHqROIUTkw6E+M3lH0PTQ==} @@ -1835,8 +1836,8 @@ packages: resolution: {integrity: sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==} engines: {node: 18 || 20 || >=22} - brace-expansion@5.0.8: - resolution: {integrity: sha512-JZyDyq3D4AUifKTPOB7DELf6XsB3WdPuNxCtob1vFXPsSXhdAiHBWJ/tJ8HAc9aH84BK+5JFZLNkJKx3G9kzQg==} + brace-expansion@5.0.9: + resolution: {integrity: sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==} engines: {node: 20 || >=22} braces@3.0.3: @@ -2307,11 +2308,11 @@ packages: dom-helpers@5.2.1: resolution: {integrity: sha512-nRCa7CK3VTrM2NmGkIy4cbK7IZlgBE/PYMn55rrXefr5xXDP0LdtfPnblFDoVdcAfslJ7or6iqAUnx0CCGIWQA==} - dompurify@3.2.7: - resolution: {integrity: sha512-WhL/YuveyGXJaerVlMYGWhvQswa7myDG17P7Vu65EWC05o8vfeNbvNf4d/BOvH99+ZW+LlQsc1GDKMa1vNK6dw==} + dompurify@3.4.13: + resolution: {integrity: sha512-2vmYIoqjze2d+kakP8S/nS5shfsl587kzwEjcGlTdiksUVgFHnFCsLYDVj/JNqJVOQZGSYBTmuycv0PodwmnMQ==} - dompurify@3.4.11: - resolution: {integrity: sha512-zhlUV12GsaRzMsf9q5M254YhA4+VuF0fG+QFqu6aYpoGlKtz+w8//jBcGVYBgQkR5GHjUomejY84AV+/uPbWdw==} + dompurify@3.4.8: + resolution: {integrity: sha512-yb1cEmaOum7wFvOCSQxyfgVlv5D47Rc30iZWoMpbDIWTnJ6grDDQyu2KFJzB2k7u0pMuJcQ1zphH//fFnw2tjQ==} dotenv@16.6.1: resolution: {integrity: sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==} @@ -2324,8 +2325,8 @@ packages: electron-to-chromium@1.5.396: resolution: {integrity: sha512-yHiw2Y3C3H9U6TMbOfoWK/BPreiOPXRfTWPBwQBoZG6/8TB6eOPnsy5oaRYuatR7Fw2SJ4kKforgufeo7fq0EQ==} - elkjs@0.11.1: - resolution: {integrity: sha512-zxxR9k+rx5ktMwT/FwyLdPCrq7xN6e4VGGHH8hA01vVYKjTFik7nHOxBnAYtrgYUB1RpAiLvA1/U2YraWxyKKg==} + elkjs@0.12.0: + resolution: {integrity: sha512-YZcKynxVxYoKIOEpywEPwCFdg+BTbxQRNf3pbwdDCvc8O3kQD8bmIwSxKU1eOTVc4Xo+VG9Te+575mlfvOrhEQ==} emoji-regex@8.0.0: resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} @@ -2383,8 +2384,8 @@ packages: resolution: {integrity: sha512-w+5mJ3GuFL+NjVtJlvydShqE1eN3h3PbI7/5LAsYJP/2qtuMXjfL2LpHSRqo4b4eSF5K/DH1JXKUAHSB2UW50g==} engines: {node: '>= 0.4'} - es-toolkit@1.49.0: - resolution: {integrity: sha512-G5iZ6Pc/FNRY/soKZHC+TxGDD83rHUDXxzaWhGCX44vAv/tMs56WMusnm/KMNK+luUPsgA9U28cGr4RDlSzL2g==} + es-toolkit@1.50.0: + resolution: {integrity: sha512-OyZKhUVvEep9ITEiwHn8GKnMRQIVqoSIX7WnRbkWgJkllCujilqP2rD0u979tkl8wqyc8ICwlc1UBVv/Sl1G6w==} es6-promise@4.2.8: resolution: {integrity: sha512-HJDGx5daxeIvxdBxvG2cb9g4tEvwIk3i8+nhX0yGrYmZUzbkdg8QbDevheDB8gd0//uPj4c1EQua8Q+MViT0/w==} @@ -2621,8 +2622,8 @@ packages: resolution: {integrity: sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==} engines: {node: '>=14'} - form-data@4.0.5: - resolution: {integrity: sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==} + form-data@4.0.6: + resolution: {integrity: sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==} engines: {node: '>= 6'} format@0.2.2: @@ -2691,8 +2692,8 @@ packages: deprecated: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me hasBin: true - globals@17.8.0: - resolution: {integrity: sha512-Zz/LMDZScFmkakeL2cTHzf+PbWKdpU3uclqkZT7TjDG58j5WPt0PpA+n9uPI24fZtlw07q0OtEi84K+umsRzqQ==} + globals@17.9.0: + resolution: {integrity: sha512-m/MvAW61QVU5VDNF1Vj8axt016h8w7L5TU1e9zlab7XIttAT2YAlCwl75K1fOqvMM9apmD7lbCIRhpfkhmxhCg==} engines: {node: '>=18'} globalthis@1.0.4: @@ -2746,8 +2747,8 @@ packages: resolution: {integrity: sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==} engines: {node: '>= 0.4'} - hasown@2.0.3: - resolution: {integrity: sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==} + hasown@2.0.4: + resolution: {integrity: sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==} engines: {node: '>= 0.4'} hast-util-from-dom@5.0.1: @@ -3096,12 +3097,12 @@ packages: resolution: {integrity: sha512-ZZow9HBI5O6EPgSJLUb8n2NKgmVWTwCvHGwFuJlMjvLFqlGG6pjirPhtdsseaLZjSibD8eegzmYpUZwoIlj2cQ==} engines: {node: '>=4.0'} - katex@0.16.45: - resolution: {integrity: sha512-pQpZbdBu7wCTmQUh7ufPmLr0pFoObnGUoL/yhtwJDgmmQpbkg/0HSVti25Fu4rmd1oCR6NGWe9vqTWuWv3GcNA==} + katex@0.16.47: + resolution: {integrity: sha512-Eeo8Ys1doU1z+x8AZsPpQu+p/QcZBI5PeOo7QGQdy2x2m0MU/hYagBbGOmXwr5KVbEfVuWv9LpnQWeehogurjg==} hasBin: true - katex@0.17.0: - resolution: {integrity: sha512-Vdw0ATsQ9V+LuegM/BTwQqV/6cTl5lbGcIrU+BCgLxyf6bo38ybOr372tuSIxir3CN720flu1meYR6XzNMwQnw==} + katex@0.18.1: + resolution: {integrity: sha512-Td8GCYSxDAoMhHOlKmCFMJ/hz5qlAAb71n66Dryw9nfCVfumLo7nhuotbvKom/XPADmrYC3O5QR71EPq4DarJQ==} hasBin: true keyv@4.5.4: @@ -3127,74 +3128,78 @@ packages: resolution: {integrity: sha512-+bT2uH4E5LGE7h/n3evcS/sQlJXCpIp6ym8OWJ5eV6+67Dsql/LaaT7qJBAt2rzfoa/5QBGBhxDix1dMt2kQKQ==} engines: {node: '>= 0.8.0'} - lightningcss-android-arm64@1.32.0: - resolution: {integrity: sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==} + lightningcss-android-arm64@1.33.0: + resolution: {integrity: sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [android] - lightningcss-darwin-arm64@1.32.0: - resolution: {integrity: sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ==} + lightningcss-darwin-arm64@1.33.0: + resolution: {integrity: sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [darwin] - lightningcss-darwin-x64@1.32.0: - resolution: {integrity: sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w==} + lightningcss-darwin-x64@1.33.0: + resolution: {integrity: sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [darwin] - lightningcss-freebsd-x64@1.32.0: - resolution: {integrity: sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig==} + lightningcss-freebsd-x64@1.33.0: + resolution: {integrity: sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [freebsd] - lightningcss-linux-arm-gnueabihf@1.32.0: - resolution: {integrity: sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw==} + lightningcss-linux-arm-gnueabihf@1.33.0: + resolution: {integrity: sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==} engines: {node: '>= 12.0.0'} cpu: [arm] os: [linux] - lightningcss-linux-arm64-gnu@1.32.0: - resolution: {integrity: sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ==} + lightningcss-linux-arm64-gnu@1.33.0: + resolution: {integrity: sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] + libc: [glibc] - lightningcss-linux-arm64-musl@1.32.0: - resolution: {integrity: sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==} + lightningcss-linux-arm64-musl@1.33.0: + resolution: {integrity: sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] + libc: [musl] - lightningcss-linux-x64-gnu@1.32.0: - resolution: {integrity: sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==} + lightningcss-linux-x64-gnu@1.33.0: + resolution: {integrity: sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] + libc: [glibc] - lightningcss-linux-x64-musl@1.32.0: - resolution: {integrity: sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==} + lightningcss-linux-x64-musl@1.33.0: + resolution: {integrity: sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] + libc: [musl] - lightningcss-win32-arm64-msvc@1.32.0: - resolution: {integrity: sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==} + lightningcss-win32-arm64-msvc@1.33.0: + resolution: {integrity: sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [win32] - lightningcss-win32-x64-msvc@1.32.0: - resolution: {integrity: sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q==} + lightningcss-win32-x64-msvc@1.33.0: + resolution: {integrity: sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [win32] - lightningcss@1.32.0: - resolution: {integrity: sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==} + lightningcss@1.33.0: + resolution: {integrity: sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==} engines: {node: '>= 12.0.0'} lines-and-columns@1.2.4: @@ -3322,8 +3327,8 @@ packages: memoize-one@6.0.0: resolution: {integrity: sha512-rkpe71W0N0c0Xz6QD0eJETuWAJGnJ9afsl1srmwPrI+yBCkge5EycXXbYRyvL29zZVUWQCY7InPRCv3GDXuZNw==} - mermaid@11.16.0: - resolution: {integrity: sha512-Zvm3kbstgdpvIJPPItlL7fppIZ3kibvc1oZIGxdvk9t6UFz6flv+Jw7FtRGKwfcI8OckmH04LqG6LlS6X4B1pA==} + mermaid@11.16.1: + resolution: {integrity: sha512-TQsq6u22fAn3rek5VOubrhKPo1g5hwC3FXUN9hiyupTckcYiGuuKGkNQrKYwGJkXUxZdojwRG46gsSCFZMDp4g==} micromark-core-commonmark@2.0.3: resolution: {integrity: sha512-RDBrHEMSxVFLg6xvnXmb1Ayr2WzLAWjeSATAoxwKYJV94TeNavgoIdA0a9ytzDSVzBy2YKFK+emCPOEibLeCrg==} @@ -3453,8 +3458,8 @@ packages: mlly@1.8.0: resolution: {integrity: sha512-l8D9ODSRWLe2KHJSifWGwBqpTZXIXTeo8mlKjY+E2HAakaTeNpqAyBZ8GSqLzHgw4XmHmC8whvpjJNMbFZN7/g==} - monaco-editor@0.55.1: - resolution: {integrity: sha512-jz4x+TJNFHwHtwuV9vA9rMujcZRb0CEilTEwG2rRSpe/A7Jdkuj8xPKttCgOh+v/lkHy7HsZ64oj+q3xoAFl9A==} + monaco-editor@0.56.0: + resolution: {integrity: sha512-sXboRm3BeBeLm938eaiyLMe0OxzfXIlZvbv4ir/jVgQy1zDhWjgmny0WoN45fuDKhCCQsYMbBJrv/A6jd8aCUg==} ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} @@ -3473,8 +3478,8 @@ packages: resolution: {integrity: sha512-dkEJPVvun4FryqBmZ5KhDo0K9iDXAwn08tMLDinNdRBNPcYEDiWYysLcc6k3mjTMlbP9KyylvRpd4wFtwrT9rw==} engines: {node: ^20.17.0 || >=22.9.0} - nanoid@3.3.16: - resolution: {integrity: sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==} + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true @@ -3590,8 +3595,8 @@ packages: package-json-from-dist@1.0.1: resolution: {integrity: sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==} - package-manager-detector@1.6.0: - resolution: {integrity: sha512-61A5ThoTiDG/C8s8UMZwSorAGwMJ0ERVGj2OjoW5pAalsNOg15+iQiPzrLJ4jhZ1HJzmC2PIHT2oEiH3R5fzNA==} + package-manager-detector@1.8.0: + resolution: {integrity: sha512-yQA4H19AmPEoMUeavPMDIe1higySl/gH/yaQrkT/s07Qp+7pp2hYz30N3z2l5BkjVkF9Ow6o0wjJamm2y7Sn0A==} parent-module@1.0.1: resolution: {integrity: sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==} @@ -3667,13 +3672,13 @@ packages: pkg-types@1.3.1: resolution: {integrity: sha512-/Jm5M4RvtBFVkKWRu2BLUTNP8/M2a+UwuAX+ae4770q1qVGtfjG+WTCupoZixokjmHiry8uI+dlY8KXYV5HVVQ==} - playwright-core@1.62.0: - resolution: {integrity: sha512-nsNRyq0r2zsG8AcRHWknc9QRA5XCueC7gWMrs+Gx2tlZn9hcl8zudfh00lhJPY1DE7NmZ6bDsT9g2yey8mXljA==} + playwright-core@1.62.1: + resolution: {integrity: sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==} engines: {node: '>=20'} hasBin: true - playwright@1.62.0: - resolution: {integrity: sha512-Z14dG305dgaLu6foB1TXQagFiW8JfSUIUaUuPaKQ6NtBPKF1P/qXcqfh6c6K/icPqdy37JmjbiBXf6JNg6Sylw==} + playwright@1.62.1: + resolution: {integrity: sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==} engines: {node: '>=20'} hasBin: true @@ -3691,8 +3696,8 @@ packages: resolution: {integrity: sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg==} engines: {node: '>= 0.4'} - postcss@8.5.20: - resolution: {integrity: sha512-lW616l85ucIQL+FocMmL7pQFPqBmwejrCMg+iPxyImlrANNJG9NHq/RkyCZopDhd8C3LA03PHRJDjkbGu8vvug==} + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} engines: {node: ^10 || ^12 || >=14} prelude-ls@1.2.1: @@ -3754,8 +3759,8 @@ packages: peerDependencies: react: ^19.2.8 - react-hook-form@7.83.0: - resolution: {integrity: sha512-AXt8cMCmx5a7u4uvpb2uRFVrWQhllI4pV+LSykxIac/hjt44TnQkmX9BKuQi2i+LDC62esmiLpilkav+kjVf/A==} + react-hook-form@7.84.0: + resolution: {integrity: sha512-+hWvQP6GLco56mDwrbU4XnHix8t1z90ltZsDIrREl+jnQFQxYLX8oAzqe/Xn8nHpmoXTY5M6oEXrAhbP1qevNQ==} engines: {node: '>=18.0.0'} peerDependencies: react: ^16.8.0 || ^17 || ^18 || ^19 @@ -3811,15 +3816,15 @@ packages: react: ^16.14.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc react-dom: ^16.14.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc - react-router-dom@7.18.1: - resolution: {integrity: sha512-KaZh+X/6UtEp28x51AUYZDMg9NGoz2ja3dNHa+ta/tk40vCzKhQ/RypCWBMLbmDr6//E24Vv5uPsrqXFozdkAg==} + react-router-dom@7.18.2: + resolution: {integrity: sha512-AIKJ/jgGlFb3EbfCXk5Gzshiwt+l3mqbCrNjmEWMMjqQxNJ3svBa6bgzFyCC2Sw3RA0VWF1kg3uQf2OFhxb8hw==} engines: {node: '>=20.0.0'} peerDependencies: react: '>=18' react-dom: '>=18' - react-router@7.18.1: - resolution: {integrity: sha512-GDLgg3i3uM0aeJO3Fm+TCS+sDQ7gu12T6x0qdTEzcwqEfleci7JwugVNIF3U//0FWKnJT7ptG+20B2jfDqnZAg==} + react-router@7.18.2: + resolution: {integrity: sha512-aUVMjFm3GAPTTZL7oYr5E7ETiqfQCHRLH+B+5afnICvf0r7kkK4eR6SMuwbSTJw/7t+12khT/Kahij49fqOCIg==} engines: {node: '>=20.0.0'} peerDependencies: react: '>=18' @@ -3925,8 +3930,8 @@ packages: robust-predicates@3.0.3: resolution: {integrity: sha512-NS3levdsRIUOmiJ8FZWCP7LG3QpJyrs/TE0Zpf1yvZu8cAJJ6QMW92H1c7kWpdIHo8RvmLxN/o2JXTKHp74lUA==} - rolldown@1.1.5: - resolution: {integrity: sha512-t9z29cJjXf/vxQ8dyhCSpt6H6aSwHTk8cT5I3iy6SMXuFpk5mB6PL6XfC8PCwrPTx93udwKUm9HRteAlTGBLiA==} + rolldown@1.2.3: + resolution: {integrity: sha512-rn9wpmxplLf7NLNyCk9FyWh3FM43DbY8jOzCdEPzH7uflhTftRbCEpqi6Ly2osgoU8OwObtmavMbWLaWy4LX7A==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true @@ -4138,6 +4143,10 @@ packages: resolution: {integrity: sha512-SHf/r48b7vOrjve9PxJo3MN5v5yuyjHvdUcrQffT3WXMUfnGmHDVbC4k3sHJaJTgZCwpUplIaAo5ANtMyp3YHg==} engines: {node: '>=18'} + tinyexec@1.3.0: + resolution: {integrity: sha512-QKAl9m8gWWGHV8jZcPeym6j+XULi6tOf1mT83WYJ4Lk2ytW/uwAWkrP0uFsdoYMdueVJ0qs26wZ+23xeB4ibNQ==} + engines: {node: '>=18'} + tinyglobby@0.2.17: resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} engines: {node: '>=12.0.0'} @@ -4330,13 +4339,13 @@ packages: peerDependencies: vite: '>8.0.0-0' - vite@8.1.5: - resolution: {integrity: sha512-7ULLwsCdYx/nRyrpiEwvqb5TFHrMVZyBt+rg/OAXT7rgj/z+DtTDyKFeLAdDkubDVDKD8jOsndmy7m55XcfUsw==} + vite@8.2.0: + resolution: {integrity: sha512-pn+CFpM0lwDeKwmOq1ZaBK/9sjorZcgqxki6MbY/jPEVd9vichIlmlD4HmQ5wdP5EgqQCFRaACBxMC7uEGc6lQ==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: '@types/node': ^20.19.0 || >=22.12.0 - '@vitejs/devtools': ^0.3.0 + '@vitejs/devtools': ^0.4.0 esbuild: '>=0.28.1' jiti: '>=1.21.0' less: ^4.0.0 @@ -4571,8 +4580,8 @@ snapshots: '@antfu/install-pkg@1.1.0': dependencies: - package-manager-detector: 1.6.0 - tinyexec: 1.2.4 + package-manager-detector: 1.8.0 + tinyexec: 1.3.0 '@apidevtools/json-schema-ref-parser@11.6.4': dependencies: @@ -4750,6 +4759,11 @@ snapshots: dependencies: '@babel/types': 7.29.7 + '@babel/parser@7.29.8': + dependencies: + '@babel/types': 7.29.8 + optional: true + '@babel/runtime@7.26.10': dependencies: regenerator-runtime: 0.14.1 @@ -4793,17 +4807,23 @@ snapshots: '@babel/helper-string-parser': 7.29.7 '@babel/helper-validator-identifier': 7.29.7 + '@babel/types@7.29.8': + dependencies: + '@babel/helper-string-parser': 7.29.7 + '@babel/helper-validator-identifier': 7.29.7 + optional: true + '@bcoe/v8-coverage@1.0.2': {} '@braintree/sanitize-url@7.1.2': {} '@chakra-ui/anatomy@2.3.4': {} - '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@ark-ui/react': 5.37.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@emotion/is-prop-valid': 1.4.0 - '@emotion/react': 11.14.0(@types/react@19.2.17)(react@19.2.8) + '@emotion/react': 11.14.0(@types/react@19.2.18)(react@19.2.8) '@emotion/serialize': 1.3.3 '@emotion/use-insertion-effect-with-fallbacks': 1.2.0(react@19.2.8) '@emotion/utils': 1.4.2 @@ -4814,22 +4834,6 @@ snapshots: '@chevrotain/types@11.1.2': {} - '@emnapi/core@1.11.1': - dependencies: - '@emnapi/wasi-threads': 1.2.2 - tslib: 2.8.1 - optional: true - - '@emnapi/runtime@1.11.1': - dependencies: - tslib: 2.8.1 - optional: true - - '@emnapi/wasi-threads@1.2.2': - dependencies: - tslib: 2.8.1 - optional: true - '@emotion/babel-plugin@11.13.5': dependencies: '@babel/helper-module-imports': 7.25.9 @@ -4862,7 +4866,7 @@ snapshots: '@emotion/memoize@0.9.0': {} - '@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8)': + '@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8)': dependencies: '@babel/runtime': 7.26.10 '@emotion/babel-plugin': 11.13.5 @@ -4874,7 +4878,7 @@ snapshots: hoist-non-react-statics: 3.3.2 react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 transitivePeerDependencies: - supports-color @@ -5001,7 +5005,7 @@ snapshots: '@iconify/types@2.0.0': {} - '@iconify/utils@3.1.3': + '@iconify/utils@3.1.4': dependencies: '@antfu/install-pkg': 1.1.0 '@iconify/types': 2.0.0 @@ -5009,30 +5013,30 @@ snapshots: '@inquirer/ansi@2.0.7': {} - '@inquirer/confirm@6.1.1(@types/node@26.1.1)': + '@inquirer/confirm@6.1.1(@types/node@26.1.2)': dependencies: - '@inquirer/core': 11.2.1(@types/node@26.1.1) - '@inquirer/type': 4.0.7(@types/node@26.1.1) + '@inquirer/core': 11.2.1(@types/node@26.1.2) + '@inquirer/type': 4.0.7(@types/node@26.1.2) optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 - '@inquirer/core@11.2.1(@types/node@26.1.1)': + '@inquirer/core@11.2.1(@types/node@26.1.2)': dependencies: '@inquirer/ansi': 2.0.7 '@inquirer/figures': 2.0.7 - '@inquirer/type': 4.0.7(@types/node@26.1.1) + '@inquirer/type': 4.0.7(@types/node@26.1.2) cli-width: 4.1.0 fast-wrap-ansi: 0.2.2 mute-stream: 3.0.0 signal-exit: 4.1.0 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@inquirer/figures@2.0.7': {} - '@inquirer/type@4.0.7(@types/node@26.1.1)': + '@inquirer/type@4.0.7(@types/node@26.1.2)': optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@internationalized/date@3.12.2': dependencies: @@ -5085,10 +5089,10 @@ snapshots: dependencies: state-local: 1.0.7 - '@monaco-editor/react@4.7.0(monaco-editor@0.55.1)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@monaco-editor/react@4.7.0(monaco-editor@0.56.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@monaco-editor/loader': 1.5.0 - monaco-editor: 0.55.1 + monaco-editor: 0.56.0 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) @@ -5101,13 +5105,6 @@ snapshots: outvariant: 1.4.3 strict-event-emitter: 0.5.1 - '@napi-rs/wasm-runtime@1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1)': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@tybys/wasm-util': 0.10.3 - optional: true - '@open-draft/deferred-promise@2.2.0': {} '@open-draft/deferred-promise@3.0.0': {} @@ -5121,73 +5118,66 @@ snapshots: '@ota-meshi/ast-token-store@0.3.0': {} - '@oxc-project/types@0.139.0': {} + '@oxc-project/types@0.143.0': {} '@pandacss/is-valid-prop@1.11.5': {} '@pkgr/core@0.3.6': {} - '@playwright/test@1.62.0': + '@playwright/test@1.62.1': dependencies: - playwright: 1.62.0 - - '@rolldown/binding-android-arm64@1.1.5': - optional: true + playwright: 1.62.1 - '@rolldown/binding-darwin-arm64@1.1.5': + '@rolldown/binding-android-arm64@1.2.3': optional: true - '@rolldown/binding-darwin-x64@1.1.5': + '@rolldown/binding-darwin-arm64@1.2.3': optional: true - '@rolldown/binding-freebsd-x64@1.1.5': + '@rolldown/binding-darwin-x64@1.2.3': optional: true - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': + '@rolldown/binding-freebsd-x64@1.2.3': optional: true - '@rolldown/binding-linux-arm64-gnu@1.1.5': + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': optional: true - '@rolldown/binding-linux-arm64-musl@1.1.5': + '@rolldown/binding-linux-arm64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-ppc64-gnu@1.1.5': + '@rolldown/binding-linux-arm64-musl@1.2.3': optional: true - '@rolldown/binding-linux-s390x-gnu@1.1.5': + '@rolldown/binding-linux-ppc64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-gnu@1.1.5': + '@rolldown/binding-linux-s390x-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-musl@1.1.5': + '@rolldown/binding-linux-x64-gnu@1.2.3': optional: true - '@rolldown/binding-openharmony-arm64@1.1.5': + '@rolldown/binding-linux-x64-musl@1.2.3': optional: true - '@rolldown/binding-wasm32-wasi@1.1.5': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@napi-rs/wasm-runtime': 1.1.6(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1) + '@rolldown/binding-openharmony-arm64@1.2.3': optional: true - '@rolldown/binding-win32-arm64-msvc@1.1.5': + '@rolldown/binding-win32-arm64-msvc@1.2.3': optional: true - '@rolldown/binding-win32-x64-msvc@1.1.5': + '@rolldown/binding-win32-x64-msvc@1.2.3': optional: true - '@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0))': + '@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.2.3)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0))': dependencies: '@babel/core': 7.29.0 picomatch: 4.0.4 - rolldown: 1.1.5 + rolldown: 1.2.3 optionalDependencies: '@babel/runtime': 7.29.7 - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) '@rolldown/pluginutils@1.0.1': {} @@ -5203,59 +5193,59 @@ snapshots: estraverse: 5.3.0 picomatch: 4.0.5 - '@swc/core-darwin-arm64@1.15.46': + '@swc/core-darwin-arm64@1.15.47': optional: true - '@swc/core-darwin-x64@1.15.46': + '@swc/core-darwin-x64@1.15.47': optional: true - '@swc/core-linux-arm-gnueabihf@1.15.46': + '@swc/core-linux-arm-gnueabihf@1.15.47': optional: true - '@swc/core-linux-arm64-gnu@1.15.46': + '@swc/core-linux-arm64-gnu@1.15.47': optional: true - '@swc/core-linux-arm64-musl@1.15.46': + '@swc/core-linux-arm64-musl@1.15.47': optional: true - '@swc/core-linux-ppc64-gnu@1.15.46': + '@swc/core-linux-ppc64-gnu@1.15.47': optional: true - '@swc/core-linux-s390x-gnu@1.15.46': + '@swc/core-linux-s390x-gnu@1.15.47': optional: true - '@swc/core-linux-x64-gnu@1.15.46': + '@swc/core-linux-x64-gnu@1.15.47': optional: true - '@swc/core-linux-x64-musl@1.15.46': + '@swc/core-linux-x64-musl@1.15.47': optional: true - '@swc/core-win32-arm64-msvc@1.15.46': + '@swc/core-win32-arm64-msvc@1.15.47': optional: true - '@swc/core-win32-ia32-msvc@1.15.46': + '@swc/core-win32-ia32-msvc@1.15.47': optional: true - '@swc/core-win32-x64-msvc@1.15.46': + '@swc/core-win32-x64-msvc@1.15.47': optional: true - '@swc/core@1.15.46(@swc/helpers@0.5.23)': + '@swc/core@1.15.47(@swc/helpers@0.5.23)': dependencies: '@swc/counter': 0.1.3 - '@swc/types': 0.1.27 + '@swc/types': 0.1.28 optionalDependencies: - '@swc/core-darwin-arm64': 1.15.46 - '@swc/core-darwin-x64': 1.15.46 - '@swc/core-linux-arm-gnueabihf': 1.15.46 - '@swc/core-linux-arm64-gnu': 1.15.46 - '@swc/core-linux-arm64-musl': 1.15.46 - '@swc/core-linux-ppc64-gnu': 1.15.46 - '@swc/core-linux-s390x-gnu': 1.15.46 - '@swc/core-linux-x64-gnu': 1.15.46 - '@swc/core-linux-x64-musl': 1.15.46 - '@swc/core-win32-arm64-msvc': 1.15.46 - '@swc/core-win32-ia32-msvc': 1.15.46 - '@swc/core-win32-x64-msvc': 1.15.46 + '@swc/core-darwin-arm64': 1.15.47 + '@swc/core-darwin-x64': 1.15.47 + '@swc/core-linux-arm-gnueabihf': 1.15.47 + '@swc/core-linux-arm64-gnu': 1.15.47 + '@swc/core-linux-arm64-musl': 1.15.47 + '@swc/core-linux-ppc64-gnu': 1.15.47 + '@swc/core-linux-s390x-gnu': 1.15.47 + '@swc/core-linux-x64-gnu': 1.15.47 + '@swc/core-linux-x64-musl': 1.15.47 + '@swc/core-win32-arm64-msvc': 1.15.47 + '@swc/core-win32-ia32-msvc': 1.15.47 + '@swc/core-win32-x64-msvc': 1.15.47 '@swc/helpers': 0.5.23 '@swc/counter@0.1.3': {} @@ -5264,7 +5254,7 @@ snapshots: dependencies: tslib: 2.8.1 - '@swc/types@0.1.27': + '@swc/types@0.1.28': dependencies: '@swc/counter': 0.1.3 @@ -5290,15 +5280,15 @@ snapshots: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - '@tanstack/react-virtual@3.14.8(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@tanstack/react-virtual@3.14.9(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: - '@tanstack/virtual-core': 3.17.6 + '@tanstack/virtual-core': 3.17.7 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) '@tanstack/table-core@8.21.3': {} - '@tanstack/virtual-core@3.17.6': {} + '@tanstack/virtual-core@3.17.7': {} '@testing-library/dom@10.4.0': dependencies: @@ -5321,15 +5311,15 @@ snapshots: picocolors: 1.1.1 redent: 3.0.0 - '@testing-library/react@16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@testing-library/react@16.3.2(@testing-library/dom@10.4.0)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@babel/runtime': 7.28.6 '@testing-library/dom': 10.4.0 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) optionalDependencies: - '@types/react': 19.2.17 - '@types/react-dom': 19.2.3(@types/react@19.2.17) + '@types/react': 19.2.18 + '@types/react-dom': 19.2.4(@types/react@19.2.18) '@trivago/prettier-plugin-sort-imports@6.0.2(prettier@3.9.6)': dependencies: @@ -5351,11 +5341,6 @@ snapshots: path-browserify: 1.0.1 tinyglobby: 0.2.17 - '@tybys/wasm-util@0.10.3': - dependencies: - tslib: 2.8.1 - optional: true - '@types/aria-query@5.0.4': {} '@types/chai@5.2.2': @@ -5415,6 +5400,10 @@ snapshots: dependencies: '@types/geojson': 7946.0.16 + '@types/d3-geo@3.1.1': + dependencies: + '@types/geojson': 7946.0.16 + '@types/d3-hierarchy@3.1.7': {} '@types/d3-interpolate@3.0.1': @@ -5431,7 +5420,7 @@ snapshots: '@types/d3-quadtree@3.0.6': {} - '@types/d3-random@3.0.3': {} + '@types/d3-random@3.0.4': {} '@types/d3-scale-chromatic@3.1.0': {} @@ -5488,13 +5477,13 @@ snapshots: '@types/d3-fetch': 3.0.7 '@types/d3-force': 3.0.10 '@types/d3-format': 3.0.4 - '@types/d3-geo': 3.1.0 + '@types/d3-geo': 3.1.1 '@types/d3-hierarchy': 3.1.7 '@types/d3-interpolate': 3.0.4 '@types/d3-path': 3.1.1 '@types/d3-polygon': 3.0.2 '@types/d3-quadtree': 3.0.6 - '@types/d3-random': 3.0.3 + '@types/d3-random': 3.0.4 '@types/d3-scale': 4.0.9 '@types/d3-scale-chromatic': 3.1.0 '@types/d3-selection': 3.0.11 @@ -5535,7 +5524,7 @@ snapshots: '@types/ms@2.1.0': {} - '@types/node@26.1.1': + '@types/node@26.1.2': dependencies: undici-types: 8.3.0 @@ -5543,25 +5532,25 @@ snapshots: '@types/prismjs@1.26.6': {} - '@types/react-dom@19.2.3(@types/react@19.2.17)': + '@types/react-dom@19.2.4(@types/react@19.2.18)': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 '@types/react-syntax-highlighter@15.5.13': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 - '@types/react-transition-group@4.4.12(@types/react@19.2.17)': + '@types/react-transition-group@4.4.12(@types/react@19.2.18)': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 - '@types/react@19.2.17': + '@types/react@19.2.18': dependencies: csstype: 3.2.3 '@types/set-cookie-parser@2.4.10': dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@types/statuses@2.0.6': {} @@ -5576,7 +5565,7 @@ snapshots: '@types/ws@8.18.1': dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@typescript-eslint/eslint-plugin@8.65.0(@typescript-eslint/parser@8.65.0(eslint@10.8.0(jiti@1.21.7))(typescript@6.0.3))(eslint@10.8.0(jiti@1.21.7))(typescript@6.0.3)': dependencies: @@ -5682,27 +5671,27 @@ snapshots: dependencies: '@visx/vendor': 4.0.0 - '@visx/group@4.0.0(@types/react@19.2.17)(react@19.2.8)': + '@visx/group@4.0.0(@types/react@19.2.18)(react@19.2.8)': dependencies: classnames: 2.5.1 react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 '@visx/scale@4.0.0': dependencies: '@visx/vendor': 4.0.0 - '@visx/shape@4.0.0(@types/react@19.2.17)(react@19.2.8)': + '@visx/shape@4.0.0(@types/react@19.2.18)(react@19.2.8)': dependencies: '@visx/curve': 4.0.0 - '@visx/group': 4.0.0(@types/react@19.2.17)(react@19.2.8) + '@visx/group': 4.0.0(@types/react@19.2.18)(react@19.2.8) '@visx/scale': 4.0.0 '@visx/vendor': 4.0.0 classnames: 2.5.1 react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 '@visx/vendor@4.0.0': dependencies: @@ -5730,20 +5719,20 @@ snapshots: d3-time-format: 4.1.0 internmap: 2.0.3 - '@vitejs/plugin-react-swc@4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0))': + '@vitejs/plugin-react-swc@4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0))': dependencies: '@rolldown/pluginutils': 1.0.1 - '@swc/core': 1.15.46(@swc/helpers@0.5.23) - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + '@swc/core': 1.15.47(@swc/helpers@0.5.23) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) transitivePeerDependencies: - '@swc/helpers' - '@vitejs/plugin-react@6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0))': + '@vitejs/plugin-react@6.0.5(@rolldown/plugin-babel@0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.2.3)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0))': dependencies: '@rolldown/pluginutils': 1.0.1 - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) optionalDependencies: - '@rolldown/plugin-babel': 0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + '@rolldown/plugin-babel': 0.2.3(@babel/core@7.29.0)(@babel/runtime@7.29.7)(rolldown@1.2.3)(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) babel-plugin-react-compiler: 1.0.0 '@vitest/coverage-v8@4.1.10(vitest@4.1.10)': @@ -5758,7 +5747,7 @@ snapshots: obug: 2.1.1 std-env: 4.0.0 tinyrainbow: 3.1.0 - vitest: 4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3))(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + vitest: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3))(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) '@vitest/expect@4.1.10': dependencies: @@ -5769,14 +5758,14 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.0 - '@vitest/mocker@4.1.10(msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3))(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0))': + '@vitest/mocker@4.1.10(msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3))(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - msw: 2.15.0(@types/node@26.1.1)(typescript@6.0.3) - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + msw: 2.15.0(@types/node@26.1.2)(typescript@6.0.3) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) '@vitest/pretty-format@4.1.10': dependencies: @@ -5802,16 +5791,16 @@ snapshots: convert-source-map: 2.0.0 tinyrainbow: 3.1.0 - '@xyflow/react@12.11.2(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@xyflow/react@12.11.2(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@xyflow/system': 0.0.79 classcat: 5.0.5 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - zustand: 4.5.7(@types/react@19.2.17)(react@19.2.8) + zustand: 4.5.7(@types/react@19.2.18)(react@19.2.8) optionalDependencies: - '@types/react': 19.2.17 - '@types/react-dom': 19.2.3(@types/react@19.2.17) + '@types/react': 19.2.18 + '@types/react-dom': 19.2.4(@types/react@19.2.18) transitivePeerDependencies: - immer @@ -6525,10 +6514,10 @@ snapshots: axe-core@4.10.3: {} - axios@1.18.1: + axios@1.19.0: dependencies: follow-redirects: 1.16.0 - form-data: 4.0.5 + form-data: 4.0.6 https-proxy-agent: 5.0.1 proxy-from-env: 2.1.0 transitivePeerDependencies: @@ -6570,7 +6559,7 @@ snapshots: dependencies: balanced-match: 4.0.4 - brace-expansion@5.0.8: + brace-expansion@5.0.9: dependencies: balanced-match: 4.0.4 @@ -6588,7 +6577,7 @@ snapshots: buffer-image-size@0.6.4: dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 builtin-modules@5.3.0: {} @@ -6636,12 +6625,12 @@ snapshots: chai@6.2.2: {} - chakra-react-select@6.1.3(@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(@types/react@19.2.17)(next-themes@0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + chakra-react-select@6.1.3(@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(@types/react@19.2.18)(next-themes@0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: - '@chakra-ui/react': 3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + '@chakra-ui/react': 3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) next-themes: 0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) react: 19.2.8 - react-select: 5.10.2(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + react-select: 5.10.2(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) transitivePeerDependencies: - '@types/react' - react-dom @@ -7052,11 +7041,11 @@ snapshots: '@babel/runtime': 7.29.7 csstype: 3.2.3 - dompurify@3.2.7: + dompurify@3.4.13: optionalDependencies: '@types/trusted-types': 2.0.7 - dompurify@3.4.11: + dompurify@3.4.8: optionalDependencies: '@types/trusted-types': 2.0.7 @@ -7070,7 +7059,7 @@ snapshots: electron-to-chromium@1.5.396: {} - elkjs@0.11.1: {} + elkjs@0.12.0: {} emoji-regex@8.0.0: {} @@ -7110,7 +7099,7 @@ snapshots: has-property-descriptors: 1.0.2 has-proto: 1.2.0 has-symbols: 1.1.0 - hasown: 2.0.3 + hasown: 2.0.2 internal-slot: 1.1.0 is-array-buffer: 3.0.5 is-callable: 1.2.7 @@ -7174,11 +7163,11 @@ snapshots: es-errors: 1.3.0 get-intrinsic: 1.3.0 has-tostringtag: 1.0.2 - hasown: 2.0.3 + hasown: 2.0.4 es-shim-unscopables@1.1.0: dependencies: - hasown: 2.0.3 + hasown: 2.0.2 es-to-primitive@1.3.0: dependencies: @@ -7186,7 +7175,7 @@ snapshots: is-date-object: 1.1.0 is-symbol: 1.1.1 - es-toolkit@1.49.0: {} + es-toolkit@1.50.0: {} es6-promise@4.2.8: {} @@ -7311,7 +7300,7 @@ snapshots: entities: 4.5.0 eslint: 10.8.0(jiti@1.21.7) find-up-simple: 1.0.1 - globals: 17.8.0 + globals: 17.9.0 indent-string: 5.0.0 is-builtin-module: 5.0.0 is-identifier: 1.1.0 @@ -7470,12 +7459,12 @@ snapshots: cross-spawn: 7.0.6 signal-exit: 4.1.0 - form-data@4.0.5: + form-data@4.0.6: dependencies: asynckit: 0.4.0 combined-stream: 1.0.8 es-set-tostringtag: 2.1.0 - hasown: 2.0.3 + hasown: 2.0.4 mime-types: 2.1.35 format@0.2.2: {} @@ -7496,7 +7485,7 @@ snapshots: call-bound: 1.0.4 define-properties: 1.2.1 functions-have-names: 1.2.3 - hasown: 2.0.3 + hasown: 2.0.2 is-callable: 1.2.7 functions-have-names@1.2.3: {} @@ -7515,7 +7504,7 @@ snapshots: get-proto: 1.0.1 gopd: 1.2.0 has-symbols: 1.1.0 - hasown: 2.0.3 + hasown: 2.0.2 math-intrinsics: 1.1.0 get-proto@1.0.1: @@ -7556,7 +7545,7 @@ snapshots: package-json-from-dist: 1.0.1 path-scurry: 2.0.2 - globals@17.8.0: {} + globals@17.9.0: {} globalthis@1.0.4: dependencies: @@ -7580,7 +7569,7 @@ snapshots: happy-dom@20.11.1: dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@types/whatwg-mimetype': 3.0.2 '@types/ws': 8.18.1 buffer-image-size: 0.6.4 @@ -7613,7 +7602,7 @@ snapshots: dependencies: function-bind: 1.1.2 - hasown@2.0.3: + hasown@2.0.4: dependencies: function-bind: 1.1.2 @@ -7771,7 +7760,7 @@ snapshots: internal-slot@1.1.0: dependencies: es-errors: 1.3.0 - hasown: 2.0.3 + hasown: 2.0.2 side-channel: 1.1.0 internmap@1.0.1: {} @@ -7822,7 +7811,7 @@ snapshots: is-core-module@2.16.1: dependencies: - hasown: 2.0.3 + hasown: 2.0.2 is-data-view@1.0.2: dependencies: @@ -7881,7 +7870,7 @@ snapshots: call-bound: 1.0.4 gopd: 1.2.0 has-tostringtag: 1.0.2 - hasown: 2.0.3 + hasown: 2.0.2 is-set@2.0.3: {} @@ -7996,11 +7985,11 @@ snapshots: object.assign: 4.1.7 object.values: 1.2.1 - katex@0.16.45: + katex@0.16.47: dependencies: commander: 8.3.0 - katex@0.17.0: + katex@0.18.1: dependencies: commander: 8.3.0 @@ -8025,54 +8014,54 @@ snapshots: prelude-ls: 1.2.1 type-check: 0.4.0 - lightningcss-android-arm64@1.32.0: + lightningcss-android-arm64@1.33.0: optional: true - lightningcss-darwin-arm64@1.32.0: + lightningcss-darwin-arm64@1.33.0: optional: true - lightningcss-darwin-x64@1.32.0: + lightningcss-darwin-x64@1.33.0: optional: true - lightningcss-freebsd-x64@1.32.0: + lightningcss-freebsd-x64@1.33.0: optional: true - lightningcss-linux-arm-gnueabihf@1.32.0: + lightningcss-linux-arm-gnueabihf@1.33.0: optional: true - lightningcss-linux-arm64-gnu@1.32.0: + lightningcss-linux-arm64-gnu@1.33.0: optional: true - lightningcss-linux-arm64-musl@1.32.0: + lightningcss-linux-arm64-musl@1.33.0: optional: true - lightningcss-linux-x64-gnu@1.32.0: + lightningcss-linux-x64-gnu@1.33.0: optional: true - lightningcss-linux-x64-musl@1.32.0: + lightningcss-linux-x64-musl@1.33.0: optional: true - lightningcss-win32-arm64-msvc@1.32.0: + lightningcss-win32-arm64-msvc@1.33.0: optional: true - lightningcss-win32-x64-msvc@1.32.0: + lightningcss-win32-x64-msvc@1.33.0: optional: true - lightningcss@1.32.0: + lightningcss@1.33.0: dependencies: detect-libc: 2.1.2 optionalDependencies: - lightningcss-android-arm64: 1.32.0 - lightningcss-darwin-arm64: 1.32.0 - lightningcss-darwin-x64: 1.32.0 - lightningcss-freebsd-x64: 1.32.0 - lightningcss-linux-arm-gnueabihf: 1.32.0 - lightningcss-linux-arm64-gnu: 1.32.0 - lightningcss-linux-arm64-musl: 1.32.0 - lightningcss-linux-x64-gnu: 1.32.0 - lightningcss-linux-x64-musl: 1.32.0 - lightningcss-win32-arm64-msvc: 1.32.0 - lightningcss-win32-x64-msvc: 1.32.0 + lightningcss-android-arm64: 1.33.0 + lightningcss-darwin-arm64: 1.33.0 + lightningcss-darwin-x64: 1.33.0 + lightningcss-freebsd-x64: 1.33.0 + lightningcss-linux-arm-gnueabihf: 1.33.0 + lightningcss-linux-arm64-gnu: 1.33.0 + lightningcss-linux-arm64-musl: 1.33.0 + lightningcss-linux-x64-gnu: 1.33.0 + lightningcss-linux-x64-musl: 1.33.0 + lightningcss-win32-arm64-msvc: 1.33.0 + lightningcss-win32-x64-msvc: 1.33.0 lines-and-columns@1.2.4: {} @@ -8111,8 +8100,8 @@ snapshots: magicast@0.3.5: dependencies: - '@babel/parser': 7.29.7 - '@babel/types': 7.29.7 + '@babel/parser': 7.29.8 + '@babel/types': 7.29.8 source-map-js: 1.2.1 optional: true @@ -8309,10 +8298,10 @@ snapshots: memoize-one@6.0.0: {} - mermaid@11.16.0: + mermaid@11.16.1: dependencies: '@braintree/sanitize-url': 7.1.2 - '@iconify/utils': 3.1.3 + '@iconify/utils': 3.1.4 '@mermaid-js/parser': 1.2.0 '@types/d3': 7.4.3 '@upsetjs/venn.js': 2.0.0 @@ -8323,9 +8312,9 @@ snapshots: d3-sankey: 0.12.3 dagre-d3-es: 7.0.14 dayjs: 1.11.21 - dompurify: 3.4.11 - es-toolkit: 1.49.0 - katex: 0.16.45 + dompurify: 3.4.13 + es-toolkit: 1.50.0 + katex: 0.16.47 khroma: 2.1.0 marked: 16.4.2 roughjs: 4.6.6 @@ -8414,7 +8403,7 @@ snapshots: dependencies: '@types/katex': 0.16.8 devlop: 1.1.0 - katex: 0.16.45 + katex: 0.16.47 micromark-factory-space: 2.0.1 micromark-util-character: 2.1.1 micromark-util-symbol: 2.0.1 @@ -8548,7 +8537,7 @@ snapshots: minimatch@10.2.6: dependencies: - brace-expansion: 5.0.8 + brace-expansion: 5.0.9 minimatch@3.1.5: dependencies: @@ -8573,16 +8562,16 @@ snapshots: pkg-types: 1.3.1 ufo: 1.6.1 - monaco-editor@0.55.1: + monaco-editor@0.56.0: dependencies: - dompurify: 3.2.7 + dompurify: 3.4.8 marked: 14.0.0 ms@2.1.3: {} - msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3): + msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3): dependencies: - '@inquirer/confirm': 6.1.1(@types/node@26.1.1) + '@inquirer/confirm': 6.1.1(@types/node@26.1.2) '@mswjs/interceptors': 0.41.9 '@open-draft/deferred-promise': 3.0.0 '@types/statuses': 2.0.6 @@ -8607,7 +8596,7 @@ snapshots: mute-stream@3.0.0: {} - nanoid@3.3.16: {} + nanoid@3.3.18: {} natural-compare@1.4.0: {} @@ -8729,7 +8718,7 @@ snapshots: package-json-from-dist@1.0.1: {} - package-manager-detector@1.6.0: {} + package-manager-detector@1.8.0: {} parent-module@1.0.1: dependencies: @@ -8801,11 +8790,11 @@ snapshots: mlly: 1.8.0 pathe: 2.0.3 - playwright-core@1.62.0: {} + playwright-core@1.62.1: {} - playwright@1.62.0: + playwright@1.62.1: dependencies: - playwright-core: 1.62.0 + playwright-core: 1.62.1 optionalDependencies: fsevents: 2.3.2 @@ -8820,9 +8809,9 @@ snapshots: possible-typed-array-names@1.1.0: {} - postcss@8.5.20: + postcss@8.5.26: dependencies: - nanoid: 3.3.16 + nanoid: 3.3.18 picocolors: 1.1.1 source-map-js: 1.2.1 @@ -8877,7 +8866,7 @@ snapshots: react: 19.2.8 scheduler: 0.27.0 - react-hook-form@7.83.0(react@19.2.8): + react-hook-form@7.84.0(react@19.2.8): dependencies: react: 19.2.8 @@ -8901,20 +8890,20 @@ snapshots: dependencies: react: 19.2.8 - react-innertext@1.1.5(@types/react@19.2.17)(react@19.2.8): + react-innertext@1.1.5(@types/react@19.2.18)(react@19.2.8): dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 react: 19.2.8 react-is@16.13.1: {} react-is@17.0.2: {} - react-markdown@10.1.0(@types/react@19.2.17)(react@19.2.8): + react-markdown@10.1.0(@types/react@19.2.18)(react@19.2.8): dependencies: '@types/hast': 3.0.4 '@types/mdast': 4.0.4 - '@types/react': 19.2.17 + '@types/react': 19.2.18 devlop: 1.1.0 hast-util-to-jsx-runtime: 2.3.6 html-url-attributes: 3.0.1 @@ -8933,13 +8922,13 @@ snapshots: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - react-router-dom@7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + react-router-dom@7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - react-router: 7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + react-router: 7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) - react-router@7.18.1(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + react-router@7.18.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: cookie: 1.1.1 react: 19.2.8 @@ -8947,19 +8936,19 @@ snapshots: optionalDependencies: react-dom: 19.2.8(react@19.2.8) - react-select@5.10.2(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8): + react-select@5.10.2(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8): dependencies: '@babel/runtime': 7.29.7 '@emotion/cache': 11.14.0 - '@emotion/react': 11.14.0(@types/react@19.2.17)(react@19.2.8) + '@emotion/react': 11.14.0(@types/react@19.2.18)(react@19.2.8) '@floating-ui/dom': 1.7.6 - '@types/react-transition-group': 4.4.12(@types/react@19.2.17) + '@types/react-transition-group': 4.4.12(@types/react@19.2.18) memoize-one: 6.0.0 prop-types: 15.8.1 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) react-transition-group: 4.4.5(react-dom@19.2.8(react@19.2.8))(react@19.2.8) - use-isomorphic-layout-effect: 1.2.1(@types/react@19.2.17)(react@19.2.8) + use-isomorphic-layout-effect: 1.2.1(@types/react@19.2.18)(react@19.2.8) transitivePeerDependencies: - '@types/react' - supports-color @@ -9033,7 +9022,7 @@ snapshots: '@types/katex': 0.16.8 hast-util-from-html-isomorphic: 2.0.0 hast-util-to-text: 4.0.2 - katex: 0.16.45 + katex: 0.16.47 unist-util-visit-parents: 6.0.1 vfile: 6.0.3 @@ -9104,26 +9093,25 @@ snapshots: robust-predicates@3.0.3: {} - rolldown@1.1.5: + rolldown@1.2.3: dependencies: - '@oxc-project/types': 0.139.0 + '@oxc-project/types': 0.143.0 '@rolldown/pluginutils': 1.0.1 optionalDependencies: - '@rolldown/binding-android-arm64': 1.1.5 - '@rolldown/binding-darwin-arm64': 1.1.5 - '@rolldown/binding-darwin-x64': 1.1.5 - '@rolldown/binding-freebsd-x64': 1.1.5 - '@rolldown/binding-linux-arm-gnueabihf': 1.1.5 - '@rolldown/binding-linux-arm64-gnu': 1.1.5 - '@rolldown/binding-linux-arm64-musl': 1.1.5 - '@rolldown/binding-linux-ppc64-gnu': 1.1.5 - '@rolldown/binding-linux-s390x-gnu': 1.1.5 - '@rolldown/binding-linux-x64-gnu': 1.1.5 - '@rolldown/binding-linux-x64-musl': 1.1.5 - '@rolldown/binding-openharmony-arm64': 1.1.5 - '@rolldown/binding-wasm32-wasi': 1.1.5 - '@rolldown/binding-win32-arm64-msvc': 1.1.5 - '@rolldown/binding-win32-x64-msvc': 1.1.5 + '@rolldown/binding-android-arm64': 1.2.3 + '@rolldown/binding-darwin-arm64': 1.2.3 + '@rolldown/binding-darwin-x64': 1.2.3 + '@rolldown/binding-freebsd-x64': 1.2.3 + '@rolldown/binding-linux-arm-gnueabihf': 1.2.3 + '@rolldown/binding-linux-arm64-gnu': 1.2.3 + '@rolldown/binding-linux-arm64-musl': 1.2.3 + '@rolldown/binding-linux-ppc64-gnu': 1.2.3 + '@rolldown/binding-linux-s390x-gnu': 1.2.3 + '@rolldown/binding-linux-x64-gnu': 1.2.3 + '@rolldown/binding-linux-x64-musl': 1.2.3 + '@rolldown/binding-openharmony-arm64': 1.2.3 + '@rolldown/binding-win32-arm64-msvc': 1.2.3 + '@rolldown/binding-win32-x64-msvc': 1.2.3 roughjs@4.6.6: dependencies: @@ -9364,6 +9352,8 @@ snapshots: tinyexec@1.2.4: {} + tinyexec@1.3.0: {} + tinyglobby@0.2.17: dependencies: fdir: 6.5.0(picomatch@4.0.5) @@ -9539,11 +9529,11 @@ snapshots: dependencies: react: 19.2.8 - use-isomorphic-layout-effect@1.2.1(@types/react@19.2.17)(react@19.2.8): + use-isomorphic-layout-effect@1.2.1(@types/react@19.2.18)(react@19.2.8): dependencies: react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 use-sync-external-store@1.6.0(react@19.2.8): dependencies: @@ -9571,27 +9561,27 @@ snapshots: '@types/unist': 3.0.3 vfile-message: 4.0.2 - vite-plugin-css-injected-by-js@5.0.2(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)): + vite-plugin-css-injected-by-js@5.0.2(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)): dependencies: - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) - vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0): + vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0): dependencies: - lightningcss: 1.32.0 + lightningcss: 1.33.0 picomatch: 4.0.5 - postcss: 8.5.20 - rolldown: 1.1.5 + postcss: 8.5.26 + rolldown: 1.2.3 tinyglobby: 0.2.17 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 fsevents: 2.3.3 jiti: 1.21.7 yaml: 2.9.0 - vitest@4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3))(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)): + vitest@4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3))(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.10 - '@vitest/mocker': 4.1.10(msw@2.15.0(@types/node@26.1.1)(typescript@6.0.3))(vite@8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0)) + '@vitest/mocker': 4.1.10(msw@2.15.0(@types/node@26.1.2)(typescript@6.0.3))(vite@8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0)) '@vitest/pretty-format': 4.1.10 '@vitest/runner': 4.1.10 '@vitest/snapshot': 4.1.10 @@ -9608,10 +9598,10 @@ snapshots: tinyexec: 1.2.4 tinyglobby: 0.2.17 tinyrainbow: 3.1.0 - vite: 8.1.5(@types/node@26.1.1)(jiti@1.21.7)(yaml@2.9.0) + vite: 8.2.0(@types/node@26.1.2)(jiti@1.21.7)(yaml@2.9.0) why-is-node-running: 2.3.0 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@vitest/coverage-v8': 4.1.10(vitest@4.1.10) happy-dom: 20.11.1 transitivePeerDependencies: @@ -9722,16 +9712,16 @@ snapshots: zod@4.3.6: {} - zustand@4.5.7(@types/react@19.2.17)(react@19.2.8): + zustand@4.5.7(@types/react@19.2.18)(react@19.2.8): dependencies: use-sync-external-store: 1.6.0(react@19.2.8) optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 react: 19.2.8 - zustand@5.0.14(@types/react@19.2.17)(react@19.2.8)(use-sync-external-store@1.6.0(react@19.2.8)): + zustand@5.0.14(@types/react@19.2.18)(react@19.2.8)(use-sync-external-store@1.6.0(react@19.2.8)): optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 react: 19.2.8 use-sync-external-store: 1.6.0(react@19.2.8) diff --git a/airflow-core/src/airflow/ui/public/i18n/README.md b/airflow-core/src/airflow/ui/public/i18n/README.md index 3c16ef0cf3574..693a79877946d 100644 --- a/airflow-core/src/airflow/ui/public/i18n/README.md +++ b/airflow-core/src/airflow/ui/public/i18n/README.md @@ -83,7 +83,7 @@ A new locale may be added when all of the following are true: The PR for a new locale should include: -- A locale-specific translation guide file at `.github/skills/airflow-translations/locales/.md` +- A locale-specific translation guide file at `.agents/skills/airflow-translations/locales/.md` - Locale files under `airflow-core/src/airflow/ui/public/i18n/locales/`. - Updates to `airflow-core/src/airflow/ui/src/i18n/config.ts`. - Updates to `dev/breeze/src/airflow_breeze/commands/ui_commands.py`. @@ -210,22 +210,22 @@ primarily intended for LLM-assisted translation work, but can also be used by hu | Locale Code | Language | Guideline File | | ----------- | ----------------------- | ------------------------------- | -| `ar` | Arabic | [.github/skills/airflow-translations/locales/ar.md](../../../../../../.github/skills/airflow-translations/locales/ar.md) | -| `ca` | Catalan | [.github/skills/airflow-translations/locales/ca.md](../../../../../../.github/skills/airflow-translations/locales/ca.md) | -| `de` | German | [.github/skills/airflow-translations/locales/de.md](../../../../../../.github/skills/airflow-translations/locales/de.md) | -| `el` | Greek | [.github/skills/airflow-translations/locales/el.md](../../../../../../.github/skills/airflow-translations/locales/el.md) | -| `es` | Spanish | [.github/skills/airflow-translations/locales/es.md](../../../../../../.github/skills/airflow-translations/locales/es.md) | -| `fr` | French | [.github/skills/airflow-translations/locales/fr.md](../../../../../../.github/skills/airflow-translations/locales/fr.md) | -| `he` | Hebrew | [.github/skills/airflow-translations/locales/he.md](../../../../../../.github/skills/airflow-translations/locales/he.md) | -| `hi` | Hindi | [.github/skills/airflow-translations/locales/hi.md](../../../../../../.github/skills/airflow-translations/locales/hi.md) | -| `hu` | Hungarian | [.github/skills/airflow-translations/locales/hu.md](../../../../../../.github/skills/airflow-translations/locales/hu.md) | -| `it` | Italian | [.github/skills/airflow-translations/locales/it.md](../../../../../../.github/skills/airflow-translations/locales/it.md) | -| `ja` | Japanese | [.github/skills/airflow-translations/locales/ja.md](../../../../../../.github/skills/airflow-translations/locales/ja.md) | -| `ko` | Korean | [.github/skills/airflow-translations/locales/ko.md](../../../../../../.github/skills/airflow-translations/locales/ko.md) | -| `nl` | Dutch | [.github/skills/airflow-translations/locales/nl.md](../../../../../../.github/skills/airflow-translations/locales/nl.md) | -| `pl` | Polish | [.github/skills/airflow-translations/locales/pl.md](../../../../../../.github/skills/airflow-translations/locales/pl.md) | -| `pt` | Portuguese | [.github/skills/airflow-translations/locales/pt.md](../../../../../../.github/skills/airflow-translations/locales/pt.md) | -| `th` | Thai | [.github/skills/airflow-translations/locales/th.md](../../../../../../.github/skills/airflow-translations/locales/th.md) | -| `tr` | Turkish | [.github/skills/airflow-translations/locales/tr.md](../../../../../../.github/skills/airflow-translations/locales/tr.md) | -| `zh-CN` | Simplified Chinese | [.github/skills/airflow-translations/locales/zh-CN.md](../../../../../../.github/skills/airflow-translations/locales/zh-CN.md) | -| `zh-TW` | Traditional Chinese | [.github/skills/airflow-translations/locales/zh-TW.md](../../../../../../.github/skills/airflow-translations/locales/zh-TW.md) | +| `ar` | Arabic | [.agents/skills/airflow-translations/locales/ar.md](../../../../../../.agents/skills/airflow-translations/locales/ar.md) | +| `ca` | Catalan | [.agents/skills/airflow-translations/locales/ca.md](../../../../../../.agents/skills/airflow-translations/locales/ca.md) | +| `de` | German | [.agents/skills/airflow-translations/locales/de.md](../../../../../../.agents/skills/airflow-translations/locales/de.md) | +| `el` | Greek | [.agents/skills/airflow-translations/locales/el.md](../../../../../../.agents/skills/airflow-translations/locales/el.md) | +| `es` | Spanish | [.agents/skills/airflow-translations/locales/es.md](../../../../../../.agents/skills/airflow-translations/locales/es.md) | +| `fr` | French | [.agents/skills/airflow-translations/locales/fr.md](../../../../../../.agents/skills/airflow-translations/locales/fr.md) | +| `he` | Hebrew | [.agents/skills/airflow-translations/locales/he.md](../../../../../../.agents/skills/airflow-translations/locales/he.md) | +| `hi` | Hindi | [.agents/skills/airflow-translations/locales/hi.md](../../../../../../.agents/skills/airflow-translations/locales/hi.md) | +| `hu` | Hungarian | [.agents/skills/airflow-translations/locales/hu.md](../../../../../../.agents/skills/airflow-translations/locales/hu.md) | +| `it` | Italian | [.agents/skills/airflow-translations/locales/it.md](../../../../../../.agents/skills/airflow-translations/locales/it.md) | +| `ja` | Japanese | [.agents/skills/airflow-translations/locales/ja.md](../../../../../../.agents/skills/airflow-translations/locales/ja.md) | +| `ko` | Korean | [.agents/skills/airflow-translations/locales/ko.md](../../../../../../.agents/skills/airflow-translations/locales/ko.md) | +| `nl` | Dutch | [.agents/skills/airflow-translations/locales/nl.md](../../../../../../.agents/skills/airflow-translations/locales/nl.md) | +| `pl` | Polish | [.agents/skills/airflow-translations/locales/pl.md](../../../../../../.agents/skills/airflow-translations/locales/pl.md) | +| `pt` | Portuguese | [.agents/skills/airflow-translations/locales/pt.md](../../../../../../.agents/skills/airflow-translations/locales/pt.md) | +| `th` | Thai | [.agents/skills/airflow-translations/locales/th.md](../../../../../../.agents/skills/airflow-translations/locales/th.md) | +| `tr` | Turkish | [.agents/skills/airflow-translations/locales/tr.md](../../../../../../.agents/skills/airflow-translations/locales/tr.md) | +| `zh-CN` | Simplified Chinese | [.agents/skills/airflow-translations/locales/zh-CN.md](../../../../../../.agents/skills/airflow-translations/locales/zh-CN.md) | +| `zh-TW` | Traditional Chinese | [.agents/skills/airflow-translations/locales/zh-TW.md](../../../../../../.agents/skills/airflow-translations/locales/zh-TW.md) | diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/en/common.json b/airflow-core/src/airflow/ui/public/i18n/locales/en/common.json index 7db7ed5306caa..235a1ff571b12 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/en/common.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/en/common.json @@ -143,6 +143,8 @@ "endTime": "End Time", "logicalDateFrom": "Logical Date From", "logicalDateTo": "Logical Date To", + "maxActiveRunsFrom": "Max Active Runs From", + "maxActiveRunsTo": "Max Active Runs To", "runAfterFrom": "Run After From", "runAfterTo": "Run After To", "searchAsset": "Search Asset", diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/en/hitl.json b/airflow-core/src/airflow/ui/public/i18n/locales/en/hitl.json index c0fd74bbfd470..a6eb3ee5d5312 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/en/hitl.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/en/hitl.json @@ -1,6 +1,7 @@ { "filters": { "body": "Body", + "completedAt": "Completed At", "createdAt": "Created At", "response": { "all": "All", diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/it/dags.json b/airflow-core/src/airflow/ui/public/i18n/locales/it/dags.json index 6c6ab552ddc9c..29258d25869ff 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/it/dags.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/it/dags.json @@ -61,6 +61,7 @@ "future": "Futuro", "onlyFailed": "Pulisci solo le task fallite", "past": "Passato", + "preventRunningTasks": "Impedisci la riesecuzione se la task è in esecuzione", "queueNew": "Inserisci nuove task", "runOnLatestVersion": "Esegui con l'ultima versione del bundle", "upstream": "Upstream" diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/th/dags.json b/airflow-core/src/airflow/ui/public/i18n/locales/th/dags.json index 9b1ce043e8f34..106ded489ec27 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/th/dags.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/th/dags.json @@ -61,6 +61,7 @@ "future": "งานในอนาคต", "onlyFailed": "ล้างเฉพาะงานที่ล้มเหลว", "past": "งานในอดีต", + "preventRunningTasks": "ป้องกันการรันซ้ำหากงานกำลังทำงาน", "queueNew": "จัดคิวงานใหม่", "runOnLatestVersion": "ทำงานด้วยเวอร์ชันล่าสุดของชุดรวม (Bundle)", "upstream": "งานก่อนหน้า" diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/assets.json b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/assets.json index 5343dbd7c5fe4..83b49e2a8998c 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/assets.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/assets.json @@ -42,7 +42,10 @@ "title": "為 {{name}} 建立資源事件" }, "events": "資源事件", - "extra": "額外資源資訊", + "filters": { + "groupPlaceholder": "搜尋群組", + "lastEventDateRange": "最後事件時間" + }, "group": "群組", "lastAssetEvent": "最後資源事件", "name": "名稱", diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/common.json b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/common.json index 84cc0ead3a62d..fe48527abf52f 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/common.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/common.json @@ -86,6 +86,7 @@ "dagRun_one": "Dag 執行", "dagRun_other": "Dag 執行", "dagRunId": "Dag 執行 ID", + "dagRunState": "Dag 執行狀態", "dagWarnings": "Dag 警告 / 錯誤", "defaultToGraphView": "預設使用圖形檢視", "defaultToGridView": "預設使用網格檢視", @@ -288,6 +289,9 @@ }, "showDetailsPanel": "顯示詳細資訊", "signedInAs": "登入身分:", + "slot": "配額", + "slot_one": "配額", + "slot_other": "配額", "source": { "hide": "隱藏來源", "hotkey": "s", diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/components.json b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/components.json index cddb3c0c73c93..54488c9372762 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/components.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/components.json @@ -7,9 +7,15 @@ "backwards": "反向執行", "dateRange": "日期範圍", "errorStartDateBeforeEndDate": "開始日期必須早於結束日期。", + "exceptionReason": { + "alreadyExists": "已存在", + "inFlight": "進行中", + "unknown": "未知" + }, "maxRuns": "活躍執行數上限", "missingAndErroredRuns": "遺漏和錯誤的執行", "missingRuns": "遺漏的執行", + "notCreatedReason": "未建立原因", "overrideExistingParams": "覆寫既有執行的參數", "permissionDenied": "試執行失敗:使用者沒有建立回填的權限。", "reprocessBehavior": "重新處理行為", @@ -29,7 +35,8 @@ "validation": { "datesRequired": "必須提供資料區間的開始與結束日期。", "startBeforeEnd": "資料區間起始日期必須早於或等於結束日期。" - } + }, + "viewSlots": "檢視回填 #{{id}} 的配額" }, "banner": { "backfillInProgress": "回填正在進行中", @@ -113,6 +120,10 @@ "location": "第 {{line}} 行,位於 {{name}}" }, "reparseDag": "重新解析 Dag", + "slowestTaskInstances": { + "empty": "此範圍內沒有已完成的任務實例", + "title": "最慢的任務實例" + }, "sortedAscending": "遞增排序", "sortedDescending": "遞減排序", "sortedUnsorted": "未排序", @@ -130,7 +141,7 @@ "triggerDag": { "button": "觸發", "dataInterval": "資料區間", - "dataIntervalAuto": "依邏輯日期與時間表推斷", + "dataIntervalAuto": "依邏輯日期與排程規則推斷", "dataIntervalManual": "手動指定", "intervalEnd": "結束", "intervalStart": "開始", diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dag.json b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dag.json index 401a19ccbd19e..95b343c55be35 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dag.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dag.json @@ -144,13 +144,7 @@ "label": "Dag 執行次數" }, "dependencies": { - "label": "相依性 (Dependencies)", - "options": { - "allDagDependencies": "所有 Dag 相依性", - "externalConditions": "外部條件", - "onlyTasks": "僅限任務" - }, - "placeholder": "相依性 (Dependencies)" + "allDagDependencies": "所有 Dag 相依性" }, "graphDirection": { "label": "圖表方向" diff --git a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dags.json b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dags.json index c13d3855dc929..b0ebeb8ea1d81 100644 --- a/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dags.json +++ b/airflow-core/src/airflow/ui/public/i18n/locales/zh-TW/dags.json @@ -17,12 +17,14 @@ "unfavorite": "未加為最愛" }, "lastRunState": "最後執行狀態", + "noTimetableTypesFound": "找不到排程規則類型", "paused": { "active": "啟用中", "all": "全部", "paused": "暫停" }, - "runIdPatternFilter": "搜尋 Dag 執行" + "runIdPatternFilter": "搜尋 Dag 執行", + "timetableType": "排程規則類型" }, "ownerLink": "擁有者 {{owner}} 的連結", "runAndTaskActions": { diff --git a/airflow-core/src/airflow/ui/src/components/MonacoEditor/configureMonaco.ts b/airflow-core/src/airflow/ui/src/components/MonacoEditor/configureMonaco.ts index 97ac79a8b36dd..3a2ff3d506bab 100644 --- a/airflow-core/src/airflow/ui/src/components/MonacoEditor/configureMonaco.ts +++ b/airflow-core/src/airflow/ui/src/components/MonacoEditor/configureMonaco.ts @@ -27,18 +27,18 @@ type MonacoEnvironment = { let configurationPromise: Promise | undefined; const loadMonacoModules = async () => { - // `editor.api` is API-only — the contribs/styles below must be side-effect imported - // to register their actions and render their glyphs. The CDN bundle pulled these in - // transitively; the local ESM build does not. + // The editor entry point is API-only; register the selected features and codicon styles + // explicitly to keep the local bundle small. const monacoApi = Promise.all([ - import("monaco-editor/esm/vs/editor/editor.api.js"), - import("monaco-editor/esm/vs/editor/contrib/folding/browser/folding.js"), - import("monaco-editor/esm/vs/editor/contrib/find/browser/findController.js"), - // monaco-editor 0.53 removed the `codiconStyles` side-effect module; import the two codicon - // stylesheets it used to pull in directly so folding/find glyphs still render. Both files - // ship in 0.52 and 0.55, so this resolves against the current pin and any newer bump. - import("monaco-editor/esm/vs/base/browser/ui/codicons/codicon/codicon.css"), - import("monaco-editor/esm/vs/base/browser/ui/codicons/codicon/codicon-modifiers.css"), + import("monaco-editor/editor"), + import("monaco-editor/features/codeAction/register"), + import("monaco-editor/features/codelens/register"), + import("monaco-editor/features/dropOrPasteInto/register"), + import("monaco-editor/features/inlayHints/register"), + import("monaco-editor/features/suggest/register"), + import("monaco-editor/features/folding/register"), + import("monaco-editor/features/find/register"), + import("monaco-editor/features/codicon/register"), ]).then(([api]) => api); // Resolve the bundled worker URLs (`?worker&url` runs the worker through Vite's worker @@ -52,17 +52,18 @@ const loadMonacoModules = async () => { // sidesteps the restriction (CORS still permits the inner import). In production the // worker is same-origin and the shim is harmless. const workerUrls = Promise.all([ - import("monaco-editor/esm/vs/editor/editor.worker.js?worker&url").then((module) => module.default), - import("monaco-editor/esm/vs/language/json/json.worker.js?worker&url").then((module) => module.default), + import("monaco-editor/editor/editor.worker.js?worker&url").then((module) => module.default), + import("monaco-editor/languages/features/json/json.worker.js?worker&url").then( + (module) => module.default, + ), ]); - // The JSON contribution registers its language as a side effect. Python is registered - // manually below from its grammar module instead of importing `python.contribution`, + // The JSON feature registers its language as a side effect. Python is registered + // manually below from its grammar module instead of importing its register module, // whose lazy tokens provider would overwrite our patched grammar on first use. - // The grammar module is a private monaco internal (verified against monaco-editor - // 0.52.2); the runtime guard below fails loudly if its export shape changes. - const jsonContribution = import("monaco-editor/esm/vs/language/json/monaco.contribution.js"); - const pythonGrammar = import("monaco-editor/esm/vs/basic-languages/python/python.js"); + // The runtime guard below fails loudly if the grammar export shape changes. + const jsonContribution = import("monaco-editor/languages/features/json/register"); + const pythonGrammar = import("monaco-editor/languages/definitions/python/python"); const [monaco, [editorWorkerUrl, jsonWorkerUrl], { conf: pythonConf, language: pythonLanguage }] = await Promise.all([monacoApi, workerUrls, pythonGrammar, jsonContribution]); diff --git a/airflow-core/src/airflow/ui/src/components/MonacoEditor/pythonFStrings.test.ts b/airflow-core/src/airflow/ui/src/components/MonacoEditor/pythonFStrings.test.ts index 1e69373e0282e..89cfb35d95954 100644 --- a/airflow-core/src/airflow/ui/src/components/MonacoEditor/pythonFStrings.test.ts +++ b/airflow-core/src/airflow/ui/src/components/MonacoEditor/pythonFStrings.test.ts @@ -127,8 +127,8 @@ describe("patchPythonFStrings (tokenized)", () => { let singleLineTokens: Array<{ offset: number; type: string }> = []; beforeAll(async () => { - const monaco = await import("monaco-editor/esm/vs/editor/editor.api.js"); - const { conf, language } = await import("monaco-editor/esm/vs/basic-languages/python/python.js"); + const monaco = await import("monaco-editor/editor"); + const { conf, language } = await import("monaco-editor/languages/definitions/python/python"); monaco.languages.register({ id: "python" }); monaco.languages.setLanguageConfiguration("python", conf); diff --git a/airflow-core/src/airflow/ui/src/components/TaskTrySelect.test.tsx b/airflow-core/src/airflow/ui/src/components/TaskTrySelect.test.tsx index 9a1e39766b11a..142dc5fc36c7d 100644 --- a/airflow-core/src/airflow/ui/src/components/TaskTrySelect.test.tsx +++ b/airflow-core/src/airflow/ui/src/components/TaskTrySelect.test.tsx @@ -18,8 +18,8 @@ */ import { ChakraProvider, defaultSystem } from "@chakra-ui/react"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; -import { render, screen } from "@testing-library/react"; -import type { PropsWithChildren } from "react"; +import { fireEvent, render, screen, waitFor } from "@testing-library/react"; +import type { PropsWithChildren, ReactNode } from "react"; import { MemoryRouter } from "react-router-dom"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -33,6 +33,12 @@ import { import { TaskTrySelect } from "./TaskTrySelect"; +vi.mock("src/components/StateBadge", () => ({ + StateBadge: ({ children, state }: { readonly children?: ReactNode; readonly state?: string | null }) => ( + {children} + ), +})); + vi.mock("src/utils", async () => { const actual = await vi.importActual("src/utils"); @@ -47,34 +53,59 @@ const DAG_RUN_ID = "test_run"; const TASK_A = "task_a"; const TASK_B = "task_b"; -const buildTaskInstance = (taskId: string, tryNumber: number): TaskInstanceResponse => +const buildTaskInstance = ( + taskId: string, + tryNumber: number, + { + mapIndex = -1, + state = "success", + }: { + readonly mapIndex?: number; + readonly state?: TaskInstanceResponse["state"]; + } = {}, +): TaskInstanceResponse => ({ dag_id: DAG_ID, dag_run_id: DAG_RUN_ID, - id: `${taskId}-id`, - map_index: -1, - state: "success", + id: `${taskId}-${mapIndex}`, + map_index: mapIndex, + state, task_display_name: taskId, task_id: taskId, try_number: tryNumber, }) as TaskInstanceResponse; -const buildTaskTry = (tryNumber: number): TaskInstanceHistoryResponse => - ({ - dag_id: DAG_ID, - dag_run_id: DAG_RUN_ID, - map_index: -1, - state: "success", - task_display_name: TASK_A, - task_id: TASK_A, - try_number: tryNumber, - }) as TaskInstanceHistoryResponse; +const buildTaskTry = ( + taskId: string, + tryNumber: number, + { + mapIndex = -1, + state = "success", + }: { + readonly mapIndex?: number; + readonly state?: TaskInstanceHistoryResponse["state"]; + } = {}, +): TaskInstanceHistoryResponse => ({ + ...buildTaskInstance(taskId, tryNumber, { mapIndex, state }), +}); -const buildTaskTries = (tryNumbers: Array): TaskInstanceHistoryCollectionResponse => ({ - task_instances: tryNumbers.map(buildTaskTry), - total_entries: tryNumbers.length, +const buildTaskTries = ( + taskInstances: Array, +): TaskInstanceHistoryCollectionResponse => ({ + task_instances: taskInstances, + total_entries: taskInstances.length, }); +const createQueryClient = () => + new QueryClient({ + defaultOptions: { + queries: { + retry: false, + staleTime: 5 * 60 * 1000, + }, + }, + }); + const createWrapper = (queryClient: QueryClient) => ({ children }: PropsWithChildren) => ( @@ -85,18 +116,31 @@ const createWrapper = ); +const expectTries = async (tries: Array) => { + await waitFor(() => { + expect( + screen + .getAllByTestId(/^log-attempt-select-button-/u) + .map((button) => button.getAttribute("data-testid")), + ).toEqual(tries.map((tryNumber) => `log-attempt-select-button-${tryNumber}`)); + }); + expect(screen.queryByTestId("log-attempt-select-button-0")).toBeNull(); +}; + +const expectTryState = (tryNumber: number, state: string) => { + expect( + screen + .getByTestId(`log-attempt-select-button-${tryNumber}`) + .querySelector("[data-state]") + ?.getAttribute("data-state"), + ).toBe(state); +}; + afterEach(() => vi.restoreAllMocks()); describe("TaskTrySelect", () => { it("refetches cached tries immediately when switching tasks", async () => { - const queryClient = new QueryClient({ - defaultOptions: { - queries: { - retry: false, - staleTime: 5 * 60 * 1000, - }, - }, - }); + const queryClient = createQueryClient(); const params = { dagId: DAG_ID, dagRunId: DAG_RUN_ID, @@ -106,9 +150,11 @@ describe("TaskTrySelect", () => { queryClient.setQueryData( UseTaskInstanceServiceGetMappedTaskInstanceTriesKeyFn(params), - buildTaskTries([1, 2]), + buildTaskTries([1, 2].map((tryNumber) => buildTaskTry(TASK_A, tryNumber))), + ); + vi.spyOn(TaskInstanceService, "getMappedTaskInstanceTries").mockResolvedValue( + buildTaskTries([1, 2, 3].map((tryNumber) => buildTaskTry(TASK_A, tryNumber))), ); - vi.spyOn(TaskInstanceService, "getMappedTaskInstanceTries").mockResolvedValue(buildTaskTries([1, 2, 3])); const { rerender } = render( , @@ -117,12 +163,102 @@ describe("TaskTrySelect", () => { rerender(); - expect(await screen.findByTestId("log-attempt-select-button-3")).toBeTruthy(); - expect( - screen - .getAllByTestId(/^log-attempt-select-button-/u) - .map((button) => button.getAttribute("data-testid")), - ).toEqual(["log-attempt-select-button-1", "log-attempt-select-button-2", "log-attempt-select-button-3"]); + await expectTries([1, 2, 3]); expect(TaskInstanceService.getMappedTaskInstanceTries).toHaveBeenCalledWith(params); }); + + it("keeps positive tries unique while switching between task instances", async () => { + const queryClient = createQueryClient(); + const histories = { + start: buildTaskTries([1, 2, 2, 3, 4].map((tryNumber) => buildTaskTry("start", tryNumber))), + task_1: buildTaskTries([ + buildTaskTry("task_1", 0, { state: "skipped" }), + buildTaskTry("task_1", 1), + buildTaskTry("task_1", 2), + ]), + task_2: buildTaskTries([buildTaskTry("task_2", 1), buildTaskTry("task_2", 2)]), + }; + + const getTries = vi + .spyOn(TaskInstanceService, "getMappedTaskInstanceTries") + .mockResolvedValue(histories.start); + + const { rerender } = render(, { + wrapper: createWrapper(queryClient), + }); + + await expectTries([1, 2, 3, 4]); + + getTries.mockResolvedValue(histories.task_1); + rerender(); + await expectTries([1, 2]); + + getTries.mockResolvedValue(histories.task_2); + rerender(); + await expectTries([1, 2]); + + getTries.mockResolvedValue(histories.task_1); + rerender(); + await expectTries([1, 2]); + + getTries.mockResolvedValue(histories.start); + rerender(); + await expectTries([1, 2, 3, 4]); + }); + + it("uses a real current try but not retry or null placeholders", async () => { + const queryClient = createQueryClient(); + const params = { + dagId: DAG_ID, + dagRunId: DAG_RUN_ID, + mapIndex: 1, + taskId: "mapped_task", + }; + const history = buildTaskTries([ + buildTaskTry("mapped_task", 1, { mapIndex: 1 }), + buildTaskTry("mapped_task", 2, { mapIndex: 1, state: "failed" }), + ]); + const onSelectTryNumber = vi.fn(); + + queryClient.setQueryData(UseTaskInstanceServiceGetMappedTaskInstanceTriesKeyFn(params), history); + vi.spyOn(TaskInstanceService, "getMappedTaskInstanceTries").mockResolvedValue(history); + + const { rerender } = render( + , + { wrapper: createWrapper(queryClient) }, + ); + + await expectTries([1, 2]); + expectTryState(2, "success"); + + rerender( + , + ); + expectTryState(2, "failed"); + + rerender( + , + ); + expectTryState(2, "failed"); + + fireEvent.click(screen.getByTestId("log-attempt-select-button-1")); + expect(onSelectTryNumber).toHaveBeenCalledOnce(); + expect(onSelectTryNumber).toHaveBeenCalledWith(1); + expect(TaskInstanceService.getMappedTaskInstanceTries).toHaveBeenCalledWith(params); + + rerender(); + expect(screen.queryByTestId(/^log-attempt-select-button-/u)).toBeNull(); + }); }); diff --git a/airflow-core/src/airflow/ui/src/components/TaskTrySelect.tsx b/airflow-core/src/airflow/ui/src/components/TaskTrySelect.tsx index 8cbf595d0c288..75132ffa013b3 100644 --- a/airflow-core/src/airflow/ui/src/components/TaskTrySelect.tsx +++ b/airflow-core/src/airflow/ui/src/components/TaskTrySelect.tsx @@ -71,11 +71,17 @@ export const TaskTrySelect = ({ onSelectTryNumber, selectedTryNumber, taskInstan const logAttemptDropdownLimit = 10; const showDropdown = finalTryNumber > logAttemptDropdownLimit; - // For some reason tries aren't sorted by try_number - const sortedTries = [...(tiHistory?.task_instances ?? [])].sort( - (tryA, tryB) => tryA.try_number - tryB.try_number, + const triesByNumber = new Map( + (tiHistory?.task_instances ?? []).filter((ti) => ti.try_number > 0).map((ti) => [ti.try_number, ti]), ); + if (finalTryNumber > 0 && state !== "up_for_retry" && state !== null) { + // The current task instance is authoritative when it is also present in history. + triesByNumber.set(finalTryNumber, taskInstance); + } + + const sortedTries = [...triesByNumber.values()].sort((tryA, tryB) => tryA.try_number - tryB.try_number); + const tryOptions = createListCollection({ items: sortedTries.map((ti) => ({ task_instance: ti, diff --git a/airflow-core/src/airflow/ui/src/components/renderStructuredLog.tsx b/airflow-core/src/airflow/ui/src/components/renderStructuredLog.tsx index a8389611f2429..bac81ef906d01 100644 --- a/airflow-core/src/airflow/ui/src/components/renderStructuredLog.tsx +++ b/airflow-core/src/airflow/ui/src/components/renderStructuredLog.tsx @@ -352,6 +352,7 @@ const renderStructuredLogImpl = ({ return ( { label: translate("components:versionDetails.bundleVersion"), type: FilterTypes.TEXT, }, + [SearchParamsKeys.COMPLETED_AT_RANGE]: { + endKey: SearchParamsKeys.COMPLETED_AT_LTE, + icon: , + label: translate("hitl:filters.completedAt"), + startKey: SearchParamsKeys.COMPLETED_AT_GTE, + type: FilterTypes.DATERANGE, + }, [SearchParamsKeys.CONF_CONTAINS]: { hotkeyDisabled: true, icon: , @@ -237,6 +245,18 @@ export const useFilterConfigs = () => { min: -1, type: FilterTypes.NUMBER, }, + [SearchParamsKeys.MAX_ACTIVE_RUNS_GTE]: { + icon: , + label: translate("common:filters.maxActiveRunsFrom"), + min: 1, + type: FilterTypes.NUMBER, + }, + [SearchParamsKeys.MAX_ACTIVE_RUNS_LTE]: { + icon: , + label: translate("common:filters.maxActiveRunsTo"), + min: 1, + type: FilterTypes.NUMBER, + }, [SearchParamsKeys.MISSED]: { icon: , label: translate("browse:deadlines.filters.status"), @@ -289,6 +309,15 @@ export const useFilterConfigs = () => { supportsAdvancedSearch: true, type: FilterTypes.TEXT, }, + [SearchParamsKeys.REPROCESS_BEHAVIOR]: { + icon: , + label: translate("components:backfill.reprocessBehavior"), + options: reprocessBehaviors.map((option) => ({ + label: translate(option.label), + value: option.value, + })), + type: FilterTypes.SELECT, + }, [SearchParamsKeys.RESPONDED_BY_USER_NAME]: { hotkeyDisabled: true, icon: , diff --git a/airflow-core/src/airflow/ui/src/constants/searchParams.ts b/airflow-core/src/airflow/ui/src/constants/searchParams.ts index b2ac553291101..df581011701e9 100644 --- a/airflow-core/src/airflow/ui/src/constants/searchParams.ts +++ b/airflow-core/src/airflow/ui/src/constants/searchParams.ts @@ -22,6 +22,9 @@ export enum SearchParamsKeys { BEFORE = "before", BODY_SEARCH = "body_search", BUNDLE_VERSION = "bundle_version", + COMPLETED_AT_GTE = "completed_at_gte", + COMPLETED_AT_LTE = "completed_at_lte", + COMPLETED_AT_RANGE = "completed_at_range", CONF_CONTAINS = "conf_contains", CONSUMING_ASSET_PATTERN = "consuming_asset_pattern", CREATED_AT_GTE = "created_at_gte", @@ -71,6 +74,9 @@ export enum SearchParamsKeys { LOGICAL_DATE_RANGE = "logical_date_range", MAP_INDEX = "map_index", MAPPED = "mapped", + MAX_ACTIVE_RUNS_GTE = "max_active_runs_gte", + MAX_ACTIVE_RUNS_LTE = "max_active_runs_lte", + MAX_ACTIVE_RUNS_RANGE = "max_active_runs_range", MISSED = "missed", NAME_PATTERN = "name_pattern", NEEDS_REVIEW = "needs_review", @@ -84,6 +90,7 @@ export enum SearchParamsKeys { POOL_NAME_PATTERN = "pool_name_pattern", QUEUE_NAME_PATTERN = "queue_name_pattern", RENDERED_MAP_INDEX = "rendered_map_index", + REPROCESS_BEHAVIOR = "reprocess_behavior", RESPONDED_BY_USER_NAME = "responded_by_user_name", RESPONSE_RECEIVED = "response_received", RETRIES = "retries", diff --git a/airflow-core/src/airflow/ui/src/i18n/config.test.ts b/airflow-core/src/airflow/ui/src/i18n/config.test.ts index 9a49276fd337a..fabde912791f6 100644 --- a/airflow-core/src/airflow/ui/src/i18n/config.test.ts +++ b/airflow-core/src/airflow/ui/src/i18n/config.test.ts @@ -17,9 +17,11 @@ * under the License. */ import { createInstance } from "i18next"; -import { describe, expect, it } from "vitest"; +import { describe, expect, it, vi } from "vitest"; -import { convertDetectedLanguage, i18nBaseOptions } from "./config"; +import { VersionService } from "openapi/requests/services.gen"; + +import { convertDetectedLanguage, i18nBaseOptions, resolveI18nVersion } from "./config"; // getBestMatchFromCodes is the resolver i18next runs on the array the // LanguageDetector returns. It is not part of i18next's public types @@ -89,3 +91,17 @@ describe("i18n language resolution", () => { expect(await resolveLanguage(["hi-IN", "en"])).toBe("hi"); }); }); + +describe("resolveI18nVersion", () => { + it("resolves to the running version", async () => { + vi.spyOn(VersionService, "getVersion").mockResolvedValueOnce({ git_version: null, version: "3.2.2" }); + + await expect(resolveI18nVersion()).resolves.toBe("3.2.2"); + }); + + it("falls back to a cache-busting value, not an empty string, when the version lookup fails", async () => { + vi.spyOn(VersionService, "getVersion").mockRejectedValueOnce(new Error("network error")); + + await expect(resolveI18nVersion()).resolves.not.toBe(""); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/i18n/config.ts b/airflow-core/src/airflow/ui/src/i18n/config.ts index 20b2970d9afc2..8b4ce747c6e69 100644 --- a/airflow-core/src/airflow/ui/src/i18n/config.ts +++ b/airflow-core/src/airflow/ui/src/i18n/config.ts @@ -144,12 +144,16 @@ const initI18n = (version: string) => { }); }; -void VersionService.getVersion() - .then((data) => { - initI18n(data.version); - }) - .catch(() => { - initI18n(""); - }); +// Falling back to an empty version on failure would drop the `?v=` cache +// buster from the translation loadPath, letting a CDN/browser keep serving a +// translation bundle cached from before the running version, indefinitely +// missing any keys added since. A timestamp isn't the true version, but it +// still busts the cache on every retry. +export const resolveI18nVersion = (): Promise => + VersionService.getVersion() + .then((data) => data.version) + .catch(() => Date.now().toString()); + +void resolveI18nVersion().then(initI18n); export { default } from "i18next"; diff --git a/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.test.tsx b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.test.tsx new file mode 100644 index 0000000000000..31ddb7e98f30d --- /dev/null +++ b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.test.tsx @@ -0,0 +1,114 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import "@testing-library/jest-dom"; +import { render, screen } from "@testing-library/react"; +import { MemoryRouter, Route, Routes } from "react-router-dom"; +import { beforeEach, describe, expect, it, vi } from "vitest"; + +import type { LightGridTaskInstanceSummary } from "openapi/requests/types.gen"; +import { TimezoneProvider } from "src/context/timezone"; +import { BaseWrapper } from "src/utils/Wrapper"; + +import { GridTI } from "./GridTI"; +import { SELECTED_TASK_OUTLINE_COLOR } from "./constants"; + +const colorModeMock = vi.fn<() => { colorMode: "dark" | "light" | undefined }>(); + +vi.mock("src/context/colorMode", () => ({ + useColorMode: () => colorModeMock(), +})); + +const taskInstance: LightGridTaskInstanceSummary = { + child_states: null, + max_end_date: null, + min_start_date: null, + state: "success", + task_display_name: "selected_task", + task_id: "selected_task", +}; + +const SELECTED_RUN_ID = "manual__2026-04-21T00:00:00+00:00"; + +const renderGridTI = (route: string, taskId = "selected_task", runId = SELECTED_RUN_ID) => + render( + + + + + + } + path="/dags/:dagId/runs/:runId/tasks/:taskId" + /> + + + + , + ); + +describe("GridTI", () => { + beforeEach(() => { + colorModeMock.mockReturnValue({ colorMode: "light" }); + }); + + it("marks the selected task square", () => { + renderGridTI(`/dags/example_dag/runs/${SELECTED_RUN_ID}/tasks/selected_task`); + + expect(screen.getByTestId("task-state-badge")).toHaveAttribute("data-selected", "true"); + expect(screen.getByTestId("task-state-badge").closest("[data-task-id='selected_task']")).toHaveAttribute( + "data-selected", + "true", + ); + }); + + it("uses a lighter outline for the selected task square in dark mode", () => { + expect(SELECTED_TASK_OUTLINE_COLOR.dark).toBe("brand.contrast"); + }); + + it("does not mark another task square as selected", () => { + renderGridTI(`/dags/example_dag/runs/${SELECTED_RUN_ID}/tasks/selected_task`, "other_task"); + + expect(screen.getByTestId("task-state-badge")).not.toHaveAttribute("data-selected"); + expect(screen.getByTestId("task-state-badge").closest("[data-task-id='other_task']")).toHaveAttribute( + "data-selected", + "false", + ); + }); + + it("keeps the task row selected without marking the same task square in another Dag run as selected", () => { + renderGridTI( + `/dags/example_dag/runs/${SELECTED_RUN_ID}/tasks/selected_task`, + "selected_task", + "other_run", + ); + + expect(screen.getByTestId("task-state-badge")).not.toHaveAttribute("data-selected"); + expect(screen.getByTestId("task-state-badge").closest("[data-task-id='selected_task']")).toHaveAttribute( + "data-selected", + "true", + ); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.tsx b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.tsx index 2a142cf98b142..4f575b8ee0ec2 100644 --- a/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.tsx +++ b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/GridTI.tsx @@ -22,9 +22,10 @@ import { Link, useLocation, useParams, useSearchParams } from "react-router-dom" import type { LightGridTaskInstanceSummary } from "openapi/requests/types.gen"; import { StateIcon } from "src/components/StateIcon"; import TaskInstanceTooltip from "src/components/TaskInstanceTooltip"; +import { useColorMode } from "src/context/colorMode"; import { buildTaskInstanceUrl } from "src/utils/links"; -import { NOTE_GRADIENT } from "./constants"; +import { NOTE_GRADIENT, SELECTED_TASK_OUTLINE_COLOR } from "./constants"; type Props = { readonly dagId: string; @@ -48,7 +49,8 @@ export const GridTI = ({ runId, taskId, }: Props) => { - const { groupId: selectedGroupId, taskId: selectedTaskId } = useParams(); + const { groupId: selectedGroupId, runId: selectedRunId, taskId: selectedTaskId } = useParams(); + const { colorMode = "light" } = useColorMode(); const location = useLocation(); const [searchParams] = useSearchParams(); @@ -68,14 +70,16 @@ export const GridTI = ({ searchParams.delete("try_number"); const redirectionSearch = searchParams.toString(); - const isSelected = selectedTaskId === taskId || selectedGroupId === taskId; + const isSelectedRow = selectedTaskId === taskId || selectedGroupId === taskId; + const isSelectedTaskInstance = selectedRunId === runId && isSelectedRow; + const selectedOutlineColor = SELECTED_TASK_OUTLINE_COLOR[colorMode]; return ( diff --git a/airflow-core/src/airflow/ui/src/layouts/Details/Grid/constants.ts b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/constants.ts index fa2f987fffbc3..1cc4d00019bea 100644 --- a/airflow-core/src/airflow/ui/src/layouts/Details/Grid/constants.ts +++ b/airflow-core/src/airflow/ui/src/layouts/Details/Grid/constants.ts @@ -41,6 +41,10 @@ export const BUNDLE_VERSION_INDICATOR_LEFT = -2; // Position from left for bundl export const BUNDLE_VERSION_ICON_SIZE = 15; // Size of the git commit icon export const DAG_VERSION_INDICATOR_HEIGHT = 104; // Height of the vertical line indicator export const VERSION_INDICATOR_Z_INDEX = 1; // Z-index for version indicators +export const SELECTED_TASK_OUTLINE_COLOR = { + dark: "brand.contrast", + light: "brand.solid", +} as const; // Render a gradient to indicate a saved note on a Dag run or task instance export const NOTE_GRADIENT = diff --git a/airflow-core/src/airflow/ui/src/mocks/handlers/backfills.ts b/airflow-core/src/airflow/ui/src/mocks/handlers/backfills.ts new file mode 100644 index 0000000000000..ee895e16987e0 --- /dev/null +++ b/airflow-core/src/airflow/ui/src/mocks/handlers/backfills.ts @@ -0,0 +1,86 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import { http, HttpResponse, type HttpHandler } from "msw"; + +const backfillBeforeFilter = { + completed_at: "2024-12-31T01:00:00Z", + created_at: "2024-12-31T00:00:00Z", + dag_display_name: "tutorial_taskflow_api", + dag_id: "tutorial_taskflow_api", + dag_run_conf: {}, + from_date: "2024-12-01T00:00:00Z", + id: 1, + is_paused: false, + max_active_runs: 10, + reprocess_behavior: "none", + to_date: "2024-12-31T00:00:00Z", + updated_at: "2024-12-31T01:00:00Z", +}; + +const backfillInRange = { + completed_at: "2025-01-16T01:00:00Z", + created_at: "2025-01-16T00:00:00Z", + dag_display_name: "tutorial_taskflow_api", + dag_id: "tutorial_taskflow_api", + dag_run_conf: {}, + from_date: "2025-01-01T00:00:00Z", + id: 2, + is_paused: false, + max_active_runs: 10, + reprocess_behavior: "none", + to_date: "2025-01-15T00:00:00Z", + updated_at: "2025-01-16T01:00:00Z", +}; + +export const handlers: Array = [ + http.get("/ui/backfills", ({ request }) => { + const url = new URL(request.url); + const fromDateGte = url.searchParams.get("from_date_gte"); + const fromDateLte = url.searchParams.get("from_date_lte"); + const toDateGte = url.searchParams.get("to_date_gte"); + const toDateLte = url.searchParams.get("to_date_lte"); + + const allBackfills = [backfillBeforeFilter, backfillInRange]; + + const filtered = allBackfills.filter((backfill) => { + const fromDate = new Date(backfill.from_date); + const toDate = new Date(backfill.to_date); + + if (fromDateGte !== null && fromDate < new Date(fromDateGte)) { + return false; + } + if (fromDateLte !== null && fromDate > new Date(fromDateLte)) { + return false; + } + if (toDateGte !== null && toDate < new Date(toDateGte)) { + return false; + } + if (toDateLte !== null && toDate > new Date(toDateLte)) { + return false; + } + + return true; + }); + + return HttpResponse.json({ + backfills: filtered, + total_entries: filtered.length, + }); + }), +]; diff --git a/airflow-core/src/airflow/ui/src/mocks/handlers/index.ts b/airflow-core/src/airflow/ui/src/mocks/handlers/index.ts index 9f60605f2b6dc..21a5ea4180442 100644 --- a/airflow-core/src/airflow/ui/src/mocks/handlers/index.ts +++ b/airflow-core/src/airflow/ui/src/mocks/handlers/index.ts @@ -16,6 +16,7 @@ * specific language governing permissions and limitations * under the License. */ +import { handlers as backfillsHandlers } from "./backfills"; import { handlers as configHandlers } from "./config"; import { handlers as dagHandlers } from "./dag"; import { handlers as dagRunsHandlers } from "./dag_runs"; @@ -23,6 +24,7 @@ import { handlers as dagsHandlers } from "./dags"; import { handlers as logHandlers } from "./log"; export const handlers = [ + ...backfillsHandlers, ...configHandlers, ...dagHandlers, ...dagRunsHandlers, diff --git a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.test.tsx b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.test.tsx index 824121bb042be..e4fe99051cbb4 100644 --- a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.test.tsx +++ b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.test.tsx @@ -29,6 +29,7 @@ import { MemoryRouter, Route, Routes } from "react-router-dom"; import { beforeEach, describe, expect, it, vi } from "vitest"; import type { BackfillDagRunResponse, BackfillResponse } from "openapi/requests/types.gen"; +import { TimezoneProvider } from "src/context/timezone"; import type * as Utils from "src/utils"; import { BaseWrapper } from "src/utils/Wrapper"; @@ -38,12 +39,14 @@ const mocks = vi.hoisted(() => ({ getBackfill: vi.fn(), listBackfillDagRuns: vi.fn(), listBackfills: vi.fn(), + listTeams: vi.fn(), })); vi.mock("openapi/queries", () => ({ useBackfillServiceGetBackfill: mocks.getBackfill, useBackfillServiceListBackfillDagRuns: mocks.listBackfillDagRuns, useBackfillServiceListBackfillsUi: mocks.listBackfills, + useTeamsServiceListTeams: mocks.listTeams, })); vi.mock("react-i18next", () => ({ @@ -126,10 +129,12 @@ const renderBackfills = (initialEntry = "/dags/example_dag/backfills") => render( - - } path="/dags/:dagId/backfills" /> - } path="/dags/:dagId/backfills/:backfillId" /> - + + + } path="/dags/:dagId/backfills" /> + } path="/dags/:dagId/backfills/:backfillId" /> + + , ); @@ -172,11 +177,17 @@ const expectDagRunsQuery = (backfillId: number) => { return options.refetchInterval; }; -describe("Backfills", () => { +describe("Backfills filters", () => { beforeEach(() => { mocks.getBackfill.mockReset(); mocks.listBackfillDagRuns.mockReset(); mocks.listBackfills.mockReset(); + mocks.listTeams.mockReset(); + mocks.listTeams.mockReturnValue({ + data: { teams: [], total_entries: 0 }, + error: undefined, + isLoading: false, + }); }); it("opens a backfill's associated slots in a dialog", async () => { diff --git a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.tsx b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.tsx index 41009ffe2e636..51a56ab7cb7ed 100644 --- a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.tsx +++ b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/Backfills.tsx @@ -20,7 +20,7 @@ import { Box, Button, Heading, Text } from "@chakra-ui/react"; import type { ColumnDef } from "@tanstack/react-table"; import type { TFunction } from "i18next"; import { useTranslation } from "react-i18next"; -import { useLocation, useNavigate, useParams } from "react-router-dom"; +import { useLocation, useNavigate, useParams, useSearchParams } from "react-router-dom"; import { useBackfillServiceListBackfillsUi } from "openapi/queries"; import type { BackfillResponse } from "openapi/requests/types.gen"; @@ -28,9 +28,25 @@ import { DataTable } from "src/components/DataTable"; import { useTableURLState } from "src/components/DataTable/useTableUrlState"; import { ErrorAlert } from "src/components/ErrorAlert"; import Time from "src/components/Time"; +import { SearchParamsKeys, type SearchParamsKeysType } from "src/constants/searchParams"; import { getDuration } from "src/utils"; import { BackfillDagRunsModal } from "./BackfillDagRunsModal"; +import { BackfillsFilters } from "./BackfillsFilters"; + +const { + COMPLETED_AT_GTE: COMPLETED_AT_GTE_PARAM, + COMPLETED_AT_LTE: COMPLETED_AT_LTE_PARAM, + CREATED_AT_GTE: CREATED_AT_GTE_PARAM, + CREATED_AT_LTE: CREATED_AT_LTE_PARAM, + END_DATE_GTE: END_DATE_GTE_PARAM, + END_DATE_LTE: END_DATE_LTE_PARAM, + MAX_ACTIVE_RUNS_GTE: MAX_ACTIVE_RUNS_GTE_PARAM, + MAX_ACTIVE_RUNS_LTE: MAX_ACTIVE_RUNS_LTE_PARAM, + REPROCESS_BEHAVIOR: REPROCESS_BEHAVIOR_PARAM, + START_DATE_GTE: START_DATE_GTE_PARAM, + START_DATE_LTE: START_DATE_LTE_PARAM, +}: SearchParamsKeysType = SearchParamsKeys; const getColumns = ( onSelectBackfill: (backfillId: number) => void, @@ -126,10 +142,38 @@ export const Backfills = () => { const { backfillId, dagId = "" } = useParams(); const selectedBackfillId = Number(backfillId); const hasSelectedBackfill = Number.isInteger(selectedBackfillId) && selectedBackfillId > 0; + + const [searchParams] = useSearchParams(); + + const startDateGte = searchParams.get(START_DATE_GTE_PARAM); + const startDateLte = searchParams.get(START_DATE_LTE_PARAM); + const endDateGte = searchParams.get(END_DATE_GTE_PARAM); + const endDateLte = searchParams.get(END_DATE_LTE_PARAM); + const createdAtGte = searchParams.get(CREATED_AT_GTE_PARAM); + const createdAtLte = searchParams.get(CREATED_AT_LTE_PARAM); + const completedAtGte = searchParams.get(COMPLETED_AT_GTE_PARAM); + const completedAtLte = searchParams.get(COMPLETED_AT_LTE_PARAM); + const maxActiveRunsGte = searchParams.get(MAX_ACTIVE_RUNS_GTE_PARAM); + const maxActiveRunsLte = searchParams.get(MAX_ACTIVE_RUNS_LTE_PARAM); + const reprocessBehavior = searchParams.get(REPROCESS_BEHAVIOR_PARAM); + const { data, error, isFetching, isLoading } = useBackfillServiceListBackfillsUi({ + completedAtGte: completedAtGte ?? undefined, + completedAtLte: completedAtLte ?? undefined, + createdAtGte: createdAtGte ?? undefined, + createdAtLte: createdAtLte ?? undefined, dagId, + fromDateGte: startDateGte ?? undefined, + fromDateLte: startDateLte ?? undefined, limit: pagination.pageSize, + maxActiveRunsGte: + maxActiveRunsGte !== null && maxActiveRunsGte !== "" ? Number(maxActiveRunsGte) : undefined, + maxActiveRunsLte: + maxActiveRunsLte !== null && maxActiveRunsLte !== "" ? Number(maxActiveRunsLte) : undefined, offset: pagination.pageIndex * pagination.pageSize, + reprocessBehavior: reprocessBehavior ?? undefined, + toDateGte: endDateGte ?? undefined, + toDateLte: endDateLte ?? undefined, }); const onSelectBackfill = (id: number) => { @@ -155,6 +199,7 @@ export const Backfills = () => { return ( + {translate("backfill", { count: data ? data.total_entries : 0 })} diff --git a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsDateFilter.test.tsx b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsDateFilter.test.tsx new file mode 100644 index 0000000000000..a02503ef93d38 --- /dev/null +++ b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsDateFilter.test.tsx @@ -0,0 +1,52 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import "@testing-library/jest-dom"; +import { render, screen, waitFor } from "@testing-library/react"; +import { describe, expect, it } from "vitest"; + +import { AppWrapper } from "src/utils/AppWrapper"; + +// The backfills mock handler (see src/mocks/handlers/backfills.ts) returns: +// - id 1 (start_date: 2024-12-01, end_date: 2024-12-31) — excluded when filtering Jan 2025 +// - id 2 (start_date: 2025-01-01, end_date: 2025-01-15) — included when filtering Jan 2025 + +const getFromDateCells = () => + screen.getAllByTestId("time-display").map((element) => element.getAttribute("datetime")); + +describe("Backfills start/end date filter", () => { + it("shows all backfills when no date filter is applied", async () => { + render(); + + await waitFor(() => expect(getFromDateCells()).toContain("2025-01-01T00:00:00Z")); + expect(getFromDateCells()).toContain("2024-12-01T00:00:00Z"); + }); + + it("filters backfills by start_date_gte and end_date_lte URL params", async () => { + render( + , + ); + + await waitFor(() => expect(getFromDateCells()).toContain("2025-01-01T00:00:00Z")); + expect(getFromDateCells()).not.toContain("2024-12-01T00:00:00Z"); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsFilters.tsx b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsFilters.tsx new file mode 100644 index 0000000000000..af2d1f31d2aaf --- /dev/null +++ b/airflow-core/src/airflow/ui/src/pages/Dag/Backfills/BackfillsFilters.tsx @@ -0,0 +1,47 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import { VStack } from "@chakra-ui/react"; + +import { FilterBar } from "src/components/FilterBar"; +import { SearchParamsKeys } from "src/constants/searchParams"; +import { useFiltersHandler, type FilterableSearchParamsKeys } from "src/utils"; + +export const BackfillsFilters = () => { + const searchParamKeys: Array = [ + SearchParamsKeys.START_DATE_RANGE, + SearchParamsKeys.END_DATE_RANGE, + SearchParamsKeys.CREATED_AT_RANGE, + SearchParamsKeys.COMPLETED_AT_RANGE, + SearchParamsKeys.MAX_ACTIVE_RUNS_GTE, + SearchParamsKeys.MAX_ACTIVE_RUNS_LTE, + SearchParamsKeys.REPROCESS_BEHAVIOR, + ]; + + const { filterConfigs, handleFiltersChange, initialValues } = useFiltersHandler(searchParamKeys); + + return ( + + + + ); +}; diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.test.tsx b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.test.tsx index f7ae3fc24aa52..0033fc2f2968c 100644 --- a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.test.tsx +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/Logs.test.tsx @@ -18,7 +18,7 @@ */ import "@testing-library/jest-dom"; import { fireEvent, render, screen, waitFor } from "@testing-library/react"; -import { describe, it, expect, beforeAll } from "vitest"; +import { beforeAll, describe, expect, it, vi } from "vitest"; import { AppWrapper } from "src/utils/AppWrapper"; @@ -438,3 +438,314 @@ describe("Task log search", () => { await expectRenderedLineNumber(/starting attempt 1 of 3/iu, 3); }, 10_000); }); + +const makeClipboardData = () => { + const store = new Map(); + + return { + getData: (type: string) => store.get(type) ?? "", + setData: (type: string, value: string) => store.set(type, value), + }; +}; + +const dispatchCopy = (clipboardData: ReturnType) => { + const copyEvent = new Event("copy", { bubbles: true, cancelable: true }); + + Object.defineProperty(copyEvent, "clipboardData", { value: clipboardData }); + document.dispatchEvent(copyEvent); + + return copyEvent; +}; + +const findRow = (text: string) => { + const container = screen.getByTestId("virtual-scroll-container"); + + return [...container.querySelectorAll("[data-index]")].find((row) => + row.textContent.includes(text), + ) as HTMLElement; +}; + +const withFakeSelection = (selection: Selection, callback: () => T): T => { + const getSelectionSpy = vi.spyOn(document, "getSelection").mockReturnValue(selection); + const result = callback(); + + getSelectionSpy.mockRestore(); + + return result; +}; + +describe("Copy across virtualized rows", () => { + it("rebuilds a removed grouped row from log data and strips line numbers", async () => { + render( + , + ); + await waitForLogs(); + + fireEvent.click(screen.getByTestId("summary-Pre Execute")); + await waitFor(() => expect(screen.getByText(/DAG bundles loaded/iu)).toBeInTheDocument()); + + const firstRow = findRow("Task started"); + const middleRow = findRow("DAG bundles loaded"); + const lastRow = findRow("Done. Returned value was: None"); + + expect(firstRow).toBeDefined(); + expect(middleRow).toBeDefined(); + expect(lastRow).toBeDefined(); + + middleRow.remove(); + + const range = document.createRange(); + + range.setStart(firstRow, 0); + range.setEnd(lastRow, lastRow.childNodes.length); + + const selection = { getRangeAt: () => range, isCollapsed: false, rangeCount: 1 } as unknown as Selection; + const clipboardData = makeClipboardData(); + const copyEvent = withFakeSelection(selection, () => dispatchCopy(clipboardData)); + + expect(copyEvent.defaultPrevented).toBe(true); + + const lines = clipboardData.getData("text/plain").split("\n"); + + expect(lines[0]).not.toMatch(/^\d/u); + expect(lines[0]).toContain("Task started"); + expect(lines).toContainEqual( + expect.stringMatching(/^\[.+\] INFO - DAG bundles loaded: dags-folder, example_dags$/u), + ); + }); + + it("rebuilds a removed ungrouped row from log data", async () => { + render( + , + ); + await waitForLogs(); + + const firstRow = findRow("Log message source details"); + const middleRow = findRow("Task started"); + const lastRow = findRow("Done. Returned value was: None"); + + expect(firstRow).toBeDefined(); + expect(middleRow).toBeDefined(); + expect(lastRow).toBeDefined(); + + middleRow.remove(); + + const range = document.createRange(); + + range.setStart(firstRow, 0); + range.setEnd(lastRow, lastRow.childNodes.length); + + const selection = { getRangeAt: () => range, isCollapsed: false, rangeCount: 1 } as unknown as Selection; + const clipboardData = makeClipboardData(); + const copyEvent = withFakeSelection(selection, () => dispatchCopy(clipboardData)); + + expect(copyEvent.defaultPrevented).toBe(true); + expect(clipboardData.getData("text/plain").split("\n")).toContainEqual( + expect.stringMatching(/^\[.+\] INFO - Task started$/u), + ); + }); + + it("leaves single-row selections to native copy", async () => { + render( + , + ); + await waitForLogs(); + + const row = findRow("Task started"); + + expect(row).toBeDefined(); + + const range = document.createRange(); + + range.setStart(row, 0); + range.setEnd(row, row.childNodes.length); + + const selection = { getRangeAt: () => range, isCollapsed: false, rangeCount: 1 } as unknown as Selection; + const clipboardData = makeClipboardData(); + const copyEvent = withFakeSelection(selection, () => dispatchCopy(clipboardData)); + + expect(copyEvent.defaultPrevented).toBe(false); + expect(clipboardData.getData("text/plain")).toBe(""); + }); +}); + +describe("Selection pinning across scrolling", () => { + it("keeps the selection-anchor row mounted after scrolling it out of the render window", async () => { + render( + , + ); + await waitForLogs(); + + fireEvent.click(screen.getByTestId("summary-Pre task execution logs")); + await waitFor(() => expect(screen.getByText(/starting attempt 1 of 3/iu)).toBeInTheDocument()); + + const anchorRow = findRow("Starting attempt 1 of 3"); + const anchorIndex = Number(anchorRow.getAttribute("data-index")); + const neighborIndex = anchorIndex + 1; + const textNode = anchorRow.querySelector("span")?.firstChild as Node; + const range = document.createRange(); + + range.setStart(textNode, 0); + range.setEnd(textNode, 0); + + const selection = { getRangeAt: () => range, isCollapsed: true, rangeCount: 1 } as unknown as Selection; + + withFakeSelection(selection, () => { + document.dispatchEvent(new Event("selectionchange")); + }); + + const container = screen.getByTestId("virtual-scroll-container"); + + fireEvent.scroll(container, { target: { scrollTop: ITEM_HEIGHT * (anchorIndex + 15) } }); + + await waitFor(() => { + expect(container.querySelector(`[data-index="${neighborIndex}"]`)).toBeNull(); + }); + expect(container.querySelector(`[data-index="${anchorIndex}"]`)).not.toBeNull(); + }); + + it("unpins once the selection is cleared", async () => { + render( + , + ); + await waitForLogs(); + + fireEvent.click(screen.getByTestId("summary-Pre task execution logs")); + await waitFor(() => expect(screen.getByText(/starting attempt 1 of 3/iu)).toBeInTheDocument()); + + const anchorRow = findRow("Starting attempt 1 of 3"); + const anchorIndex = Number(anchorRow.getAttribute("data-index")); + const textNode = anchorRow.querySelector("span")?.firstChild as Node; + const range = document.createRange(); + + range.setStart(textNode, 0); + range.setEnd(textNode, 0); + + const selection = { getRangeAt: () => range, isCollapsed: true, rangeCount: 1 } as unknown as Selection; + + withFakeSelection(selection, () => { + document.dispatchEvent(new Event("selectionchange")); + }); + + const noSelection = null as unknown as Selection; + + withFakeSelection(noSelection, () => { + document.dispatchEvent(new Event("selectionchange")); + }); + + const container = screen.getByTestId("virtual-scroll-container"); + + fireEvent.scroll(container, { target: { scrollTop: ITEM_HEIGHT * (anchorIndex + 15) } }); + + await waitFor(() => { + expect(container.querySelector(`[data-index="${anchorIndex}"]`)).toBeNull(); + }); + }); +}); + +describe("Downward drag selection", () => { + it("coalesces events and extends the selection to the mounted bottom row", async () => { + render( + , + ); + await waitForLogs(); + + const container = screen.getByTestId("virtual-scroll-container"); + const rows = container.querySelectorAll("[data-index]"); + const anchorRow = rows[0] as HTMLElement; + const lastRow = rows[rows.length - 1] as HTMLElement; + const extend = vi.fn(); + const range = document.createRange(); + + range.selectNodeContents(anchorRow); + + const selection = { + anchorNode: anchorRow, + extend, + focusNode: anchorRow, + focusOffset: 0, + getRangeAt: () => range, + rangeCount: 1, + } as unknown as Selection; + const animationFrames = new Array(); + const getSelectionSpy = vi.spyOn(document, "getSelection").mockReturnValue(selection); + const requestAnimationFrameSpy = vi + .spyOn(globalThis, "requestAnimationFrame") + .mockImplementation((callback) => { + animationFrames.push(callback); + + return animationFrames.length; + }); + const cancelAnimationFrameSpy = vi + .spyOn(globalThis, "cancelAnimationFrame") + .mockImplementation(() => undefined); + + container.getBoundingClientRect = () => ({ bottom: 500 }) as DOMRect; + lastRow.getBoundingClientRect = () => ({ bottom: 480 }) as DOMRect; + + fireEvent.pointerDown(anchorRow, { button: 0, clientY: 200, pointerType: "mouse" }); + fireEvent.pointerMove(document, { clientY: 490, pointerType: "mouse" }); + document.dispatchEvent(new Event("selectionchange")); + fireEvent.scroll(container); + + expect(animationFrames).toHaveLength(1); + animationFrames.shift()?.(0); + expect(extend).toHaveBeenCalledWith(lastRow, lastRow.childNodes.length); + + fireEvent.scroll(container); + expect(animationFrames).toHaveLength(1); + animationFrames.shift()?.(1); + expect(extend).toHaveBeenCalledTimes(2); + + fireEvent.scroll(container); + const pendingAnimationFrame = animationFrames.shift(); + + fireEvent.pointerUp(document, { pointerType: "mouse" }); + expect(cancelAnimationFrameSpy).toHaveBeenCalledWith(1); + + pendingAnimationFrame?.(2); + expect(extend).toHaveBeenCalledTimes(2); + + getSelectionSpy.mockRestore(); + requestAnimationFrameSpy.mockRestore(); + cancelAnimationFrameSpy.mockRestore(); + }); + + it("only activates for downward primary-mouse drags starting in a log row", async () => { + render( + , + ); + await waitForLogs(); + + const container = screen.getByTestId("virtual-scroll-container"); + const anchorRow = container.querySelector("[data-index]") as HTMLElement; + const rows = container.querySelectorAll("[data-index]"); + const lastRow = rows[rows.length - 1] as HTMLElement; + const requestAnimationFrameSpy = vi + .spyOn(globalThis, "requestAnimationFrame") + .mockImplementation(() => 1); + + container.getBoundingClientRect = () => ({ bottom: 500 }) as DOMRect; + lastRow.getBoundingClientRect = () => ({ bottom: 700 }) as DOMRect; + + fireEvent.pointerDown(container, { button: 0, clientY: 200, pointerType: "mouse" }); + fireEvent.pointerMove(document, { clientY: 600, pointerType: "mouse" }); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + fireEvent.pointerDown(anchorRow, { button: 2, clientY: 200, pointerType: "mouse" }); + fireEvent.pointerMove(document, { clientY: 600, pointerType: "mouse" }); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + fireEvent.pointerDown(anchorRow, { button: 0, clientY: 200, pointerType: "touch" }); + fireEvent.pointerMove(document, { clientY: 600, pointerType: "touch" }); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + fireEvent.pointerDown(anchorRow, { button: 0, clientY: 200, pointerType: "mouse" }); + fireEvent.pointerMove(document, { clientY: 50, pointerType: "mouse" }); + fireEvent.scroll(container); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + fireEvent.pointerUp(document, { pointerType: "mouse" }); + requestAnimationFrameSpy.mockRestore(); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx index 8f57b096e3cd1..d71a1e8883ceb 100644 --- a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/TaskLogContent.tsx @@ -17,7 +17,8 @@ * under the License. */ import { Box, Code, VStack } from "@chakra-ui/react"; -import { useVirtualizer } from "@tanstack/react-virtual"; +import { defaultRangeExtractor, useVirtualizer } from "@tanstack/react-virtual"; +import type { Range as VirtualizerRange } from "@tanstack/react-virtual"; import { useLayoutEffect, useRef, useCallback, useEffect } from "react"; import { ErrorAlert } from "src/components/ErrorAlert"; @@ -28,8 +29,16 @@ import type { ParsedLogEntry } from "src/queries/useLogs"; import { HighlightedText } from "./HighlightedText"; import { ScrollToButton } from "./ScrollToButton"; +import { + getBottomDragBoundary, + getBottomDragClampTarget, + extractSelectedLogText, + getEntryText, + getSelectionPinnedRows, + mergePinnedIndexes, +} from "./logSelection"; import { useLogGroups } from "./useLogGroups"; -import { getHighlightColor, scrollToBottom, scrollToTop } from "./utils"; +import { getHighlightColor, isSelectionWithin, scrollToBottom, scrollToTop } from "./utils"; export type TaskLogContentProps = { readonly currentMatchLineIndex?: number; @@ -43,8 +52,10 @@ export type TaskLogContentProps = { readonly wrap: boolean; }; -// How close to the bottom (in px) before we consider the user "at the bottom" -const SCROLL_BOTTOM_THRESHOLD = 100; +// How close to the very end (in px) the user must be for the log to keep +// following new lines. Small so that scrolling up even a little to read +// stops the follow; returning to the end resumes it. +const SCROLL_BOTTOM_THRESHOLD = 40; export const TaskLogContent = ({ currentMatchLineIndex, @@ -73,12 +84,21 @@ export const TaskLogContent = ({ const isAtBottomRef = useRef(true); const prevVisibleCountRef = useRef(0); + const pinnedRowsRef = useRef>([]); + const isSelectingRef = useRef(false); + // NaN disables clamping between drags. + const lastPointerYRef = useRef(Number.NaN); + const dragClampRafRef = useRef(0); + + const rangeExtractor = (range: VirtualizerRange) => + mergePinnedIndexes(defaultRangeExtractor(range), pinnedRowsRef.current, range.count); const rowVirtualizer = useVirtualizer({ count: visibleItems.length, estimateSize: () => 20, getScrollElement: () => parentRef.current, overscan: 10, + rangeExtractor, }); const contentHeight = rowVirtualizer.getTotalSize(); @@ -102,14 +122,134 @@ export const TaskLogContent = ({ return () => el?.removeEventListener("scroll", handleScroll); }, [handleScroll]); + useEffect(() => { + const container = parentRef.current; + + if (!container) { + return undefined; + } + const clampSelectionToBottom = () => { + dragClampRafRef.current = 0; + + if (!isSelectingRef.current) { + return; + } + const selection = document.getSelection(); + + if (!selection) { + return; + } + const clampTarget = getBottomDragClampTarget({ + container, + pointerY: lastPointerYRef.current, + selection, + }); + + if (clampTarget) { + selection.extend(clampTarget.node, clampTarget.offset); + } + }; + const scheduleBottomClamp = () => { + if (!isSelectingRef.current || dragClampRafRef.current !== 0) { + return; + } + const boundary = getBottomDragBoundary(container); + + if (boundary === undefined || lastPointerYRef.current < boundary.y) { + return; + } + dragClampRafRef.current = requestAnimationFrame(clampSelectionToBottom); + }; + const handleSelectionChange = () => { + const selection = document.getSelection(); + + pinnedRowsRef.current = getSelectionPinnedRows(selection, container); + scheduleBottomClamp(); + }; + const handlePointerDown = (event: PointerEvent) => { + const target = event.target instanceof Element ? event.target.closest("[data-index]") : null; + + if (event.button !== 0 || event.pointerType !== "mouse" || !target || !container.contains(target)) { + return; + } + isSelectingRef.current = true; + lastPointerYRef.current = event.clientY; + }; + const stopSelecting = () => { + isSelectingRef.current = false; + lastPointerYRef.current = Number.NaN; + cancelAnimationFrame(dragClampRafRef.current); + dragClampRafRef.current = 0; + }; + const handlePointerMove = (event: PointerEvent) => { + if (!isSelectingRef.current) { + return; + } + lastPointerYRef.current = event.clientY; + scheduleBottomClamp(); + }; + + container.addEventListener("pointerdown", handlePointerDown); + container.addEventListener("scroll", scheduleBottomClamp, { passive: true }); + document.addEventListener("selectionchange", handleSelectionChange); + document.addEventListener("pointermove", handlePointerMove, { passive: true }); + document.addEventListener("pointerup", stopSelecting); + document.addEventListener("pointercancel", stopSelecting); + globalThis.addEventListener("blur", stopSelecting); + + return () => { + container.removeEventListener("pointerdown", handlePointerDown); + container.removeEventListener("scroll", scheduleBottomClamp); + document.removeEventListener("selectionchange", handleSelectionChange); + document.removeEventListener("pointermove", handlePointerMove); + document.removeEventListener("pointerup", stopSelecting); + document.removeEventListener("pointercancel", stopSelecting); + globalThis.removeEventListener("blur", stopSelecting); + cancelAnimationFrame(dragClampRafRef.current); + }; + }, []); + + useEffect(() => { + const handleCopy = (event: ClipboardEvent) => { + const container = parentRef.current; + const selection = document.getSelection(); + + if (!container || !selection || !event.clipboardData) { + return; + } + const text = extractSelectedLogText({ + container, + getRowText: (index) => { + const entry = visibleItems[index]?.entry; + + return entry ? getEntryText(entry) : ""; + }, + selection, + }); + + if (text === undefined) { + return; + } + event.preventDefault(); + event.clipboardData.setData("text/plain", text); + }; + + document.addEventListener("copy", handleCopy); + + return () => document.removeEventListener("copy", handleCopy); + }, [visibleItems]); + useLayoutEffect(() => { if (visibleItems.length === 0) { return; } const isFirstLoad = prevVisibleCountRef.current === 0; const hasNewLines = visibleItems.length > prevVisibleCountRef.current; + // Pause following while the user is selecting text in the log — scrolling + // would move the text out from under the cursor and clear the selection. + const isSelecting = isSelectionWithin(document.getSelection(), parentRef.current); - if ((isFirstLoad || (hasNewLines && isAtBottomRef.current)) && !location.hash) { + if ((isFirstLoad || (hasNewLines && isAtBottomRef.current && !isSelecting)) && !location.hash) { rowVirtualizer.scrollToIndex(visibleItems.length - 1, { align: "end" }); } prevVisibleCountRef.current = visibleItems.length; diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.test.ts b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.test.ts new file mode 100644 index 0000000000000..72b10f4821735 --- /dev/null +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.test.ts @@ -0,0 +1,564 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import { createElement } from "react"; +import { afterEach, describe, expect, it } from "vitest"; + +import { + extractSelectedLogText, + getBottomDragClampTarget, + getEntryText, + getSelectionPinnedRows, + getSelectionRowRange, + mergePinnedIndexes, +} from "./logSelection"; + +const buildLogContainer = (rows: Array<{ index: number; text: string }>): HTMLElement => { + const container = document.createElement("div"); + + rows.forEach(({ index, text }) => { + const row = document.createElement("div"); + + row.setAttribute("data-index", String(index)); + row.textContent = text; + container.append(row); + }); + document.body.append(container); + + return container; +}; + +const getRowTextNode = (container: HTMLElement, index: number): Node => + container.querySelector(`[data-index="${index}"]`)?.firstChild as Node; + +const makeSelection = (range: Range): Selection => + ({ + getRangeAt: () => range, + isCollapsed: range.collapsed, + rangeCount: 1, + }) as unknown as Selection; + +type SelectBetweenOptions = { + end: Node; + endOffset: number; + start: Node; + startOffset: number; +}; + +const selectBetween = ({ end, endOffset, start, startOffset }: SelectBetweenOptions): Selection => { + const range = document.createRange(); + + range.setStart(start, startOffset); + range.setEnd(end, endOffset); + + return makeSelection(range); +}; + +afterEach(() => { + document.body.innerHTML = ""; +}); + +describe("getSelectionRowRange", () => { + it("returns the row range when both boundaries are inside log rows", () => { + const container = buildLogContainer([ + { index: 2, text: "line 2" }, + { index: 7, text: "line 7" }, + ]); + const selection = selectBetween({ + end: getRowTextNode(container, 7), + endOffset: 3, + start: getRowTextNode(container, 2), + startOffset: 1, + }); + + expect(getSelectionRowRange(selection, container)).toEqual({ end: 7, start: 2 }); + }); + + it("returns undefined for a collapsed selection", () => { + const container = buildLogContainer([{ index: 0, text: "line 0" }]); + const node = getRowTextNode(container, 0); + + expect( + getSelectionRowRange( + selectBetween({ end: node, endOffset: 2, start: node, startOffset: 2 }), + container, + ), + ).toBeUndefined(); + }); + + it("returns undefined when a boundary is outside the container", () => { + const container = buildLogContainer([{ index: 0, text: "line 0" }]); + const outside = document.createElement("div"); + + outside.setAttribute("data-index", "99"); + outside.textContent = "not a log line"; + document.body.append(outside); + + const selection = selectBetween({ + end: getRowTextNode(container, 0), + endOffset: 3, + start: outside.firstChild as Node, + startOffset: 0, + }); + + expect(getSelectionRowRange(selection, container)).toBeUndefined(); + }); + + it("returns undefined when a boundary degraded to the container itself", () => { + const container = buildLogContainer([ + { index: 0, text: "line 0" }, + { index: 1, text: "line 1" }, + ]); + const range = document.createRange(); + + range.setStart(container, 0); + range.setEnd(getRowTextNode(container, 1), 3); + + expect(getSelectionRowRange(makeSelection(range), container)).toBeUndefined(); + }); + + it("returns undefined for a null selection", () => { + const container = buildLogContainer([{ index: 0, text: "line 0" }]); + + expect(getSelectionRowRange(null, container)).toBeUndefined(); + }); +}); + +describe("getSelectionPinnedRows", () => { + it("pins both rows when both boundaries are inside log rows", () => { + const container = buildLogContainer([ + { index: 2, text: "line 2" }, + { index: 7, text: "line 7" }, + ]); + const selection = selectBetween({ + end: getRowTextNode(container, 7), + endOffset: 3, + start: getRowTextNode(container, 2), + startOffset: 1, + }); + + expect(getSelectionPinnedRows(selection, container)).toEqual([2, 7]); + }); + + it("keeps the anchor row pinned when the drag focus leaves the rows", () => { + const toolbar = document.createElement("div"); + + toolbar.textContent = "search toolbar"; + document.body.prepend(toolbar); + + const container = buildLogContainer([{ index: 100, text: "anchor line" }]); + + const selection = selectBetween({ + end: getRowTextNode(container, 100), + endOffset: 5, + start: toolbar.firstChild as Node, + startOffset: 0, + }); + + expect(getSelectionPinnedRows(selection, container)).toEqual([100]); + }); + + it("pins only the mapped row when one boundary sits on the container padding", () => { + const container = buildLogContainer([ + { index: 0, text: "line 0" }, + { index: 5, text: "line 5" }, + ]); + const range = document.createRange(); + + range.setStart(container, 0); + range.setEnd(getRowTextNode(container, 5), 3); + + expect(getSelectionPinnedRows(makeSelection(range), container)).toEqual([5]); + }); + + it("pins the caret row for a collapsed selection so shift-click extension survives scrolling", () => { + const container = buildLogContainer([{ index: 3, text: "caret line" }]); + const node = getRowTextNode(container, 3); + + expect( + getSelectionPinnedRows( + selectBetween({ end: node, endOffset: 2, start: node, startOffset: 2 }), + container, + ), + ).toEqual([3, 3]); + }); + + it("returns no pins for a null selection", () => { + const container = buildLogContainer([{ index: 0, text: "line 0" }]); + + expect(getSelectionPinnedRows(null, container)).toEqual([]); + }); +}); + +const makeDirectionalSelection = (options: { + anchor: Node; + anchorOffset: number; + focus: Node; + focusOffset: number; +}): Selection => + ({ + anchorNode: options.anchor, + anchorOffset: options.anchorOffset, + focusNode: options.focus, + focusOffset: options.focusOffset, + isCollapsed: false, + rangeCount: 1, + }) as unknown as Selection; + +const buildClampContainer = ({ + containerBottom = 500, + lastRowBottom = 700, +}: { containerBottom?: number; lastRowBottom?: number } = {}) => { + const container = buildLogContainer([ + { index: 10, text: "row ten" }, + { index: 11, text: "row eleven" }, + { index: 12, text: "row twelve" }, + ]); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + container.getBoundingClientRect = () => ({ bottom: containerBottom, top: 100 }) as unknown as DOMRect; + lastRow.getBoundingClientRect = () => ({ bottom: lastRowBottom }) as DOMRect; + + return container; +}; + +describe("getBottomDragClampTarget", () => { + it("returns undefined when there is no selection range", () => { + const container = buildClampContainer(); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + + Object.defineProperty(selection, "rangeCount", { value: 0 }); + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toBeUndefined(); + }); + + it("clamps to the last row when the pointer is below and the focus flipped above the anchor", () => { + const container = buildClampContainer(); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toEqual({ + node: lastRow, + offset: lastRow.childNodes.length, + }); + }); + + it("clamps inside the viewer after the pointer passes the last mounted row", () => { + const container = buildClampContainer({ lastRowBottom: 480 }); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + expect(getBottomDragClampTarget({ container, pointerY: 490, selection })).toEqual({ + node: lastRow, + offset: lastRow.childNodes.length, + }); + }); + + it("does not clamp inside the viewer while the last mounted row continues below it", () => { + const container = buildClampContainer(); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + + expect(getBottomDragClampTarget({ container, pointerY: 490, selection })).toBeUndefined(); + }); + + it("clamps to the last row when the pointer is below and the focus left the rows", () => { + const container = buildClampContainer(); + const outside = document.createElement("div"); + + outside.textContent = "outside"; + document.body.append(outside); + + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: outside.firstChild as Node, + focusOffset: 0, + }); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toEqual({ + node: lastRow, + offset: lastRow.childNodes.length, + }); + }); + + it("does not clamp an upward selection when the pointer is above the container", () => { + const container = buildClampContainer(); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 11), + anchorOffset: 2, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + + expect(getBottomDragClampTarget({ container, pointerY: 50, selection })).toBeUndefined(); + }); + + it("clamps a same-row focus inversion when the pointer is below the container", () => { + const container = buildClampContainer(); + const node = getRowTextNode(container, 10); + const selection = makeDirectionalSelection({ + anchor: node, + anchorOffset: 4, + focus: node, + focusOffset: 0, + }); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toEqual({ + node: lastRow, + offset: lastRow.childNodes.length, + }); + }); + + it("follows the mounted edge for a forward selection while the pointer is below the container", () => { + const container = buildClampContainer(); + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 10), + anchorOffset: 2, + focus: getRowTextNode(container, 11), + focusOffset: 3, + }); + const lastRow = container.querySelector('[data-index="12"]') as Element; + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toEqual({ + node: lastRow, + offset: lastRow.childNodes.length, + }); + }); + + it("returns undefined when the focus already sits at the clamp target", () => { + const container = buildClampContainer(); + const lastRow = container.querySelector('[data-index="12"]') as Element; + const selection = makeDirectionalSelection({ + anchor: getRowTextNode(container, 12), + anchorOffset: 2, + focus: lastRow, + focusOffset: lastRow.childNodes.length, + }); + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toBeUndefined(); + }); + + it("returns undefined when the anchor is not inside a row", () => { + const container = buildClampContainer(); + const outside = document.createElement("div"); + + outside.textContent = "outside"; + document.body.append(outside); + + const selection = makeDirectionalSelection({ + anchor: outside.firstChild as Node, + anchorOffset: 0, + focus: getRowTextNode(container, 10), + focusOffset: 0, + }); + + expect(getBottomDragClampTarget({ container, pointerY: 600, selection })).toBeUndefined(); + }); +}); + +describe("mergePinnedIndexes", () => { + it("returns the default range untouched when there is nothing to pin", () => { + expect(mergePinnedIndexes([5, 6, 7], [], 10)).toEqual([5, 6, 7]); + }); + + it("merges pinned indexes into the range, sorted and deduplicated", () => { + expect(mergePinnedIndexes([5, 6, 7], [12, 2, 6], 20)).toEqual([2, 5, 6, 7, 12]); + }); + + it("drops pinned indexes outside [0, count)", () => { + expect(mergePinnedIndexes([5, 6], [-1, 99], 10)).toEqual([5, 6]); + expect(mergePinnedIndexes([5, 6], [-1, 2, 99], 10)).toEqual([2, 5, 6]); + }); +}); + +describe("extractSelectedLogText", () => { + it("returns undefined for a selection within a single row", () => { + const container = buildLogContainer([{ index: 0, text: "hello world" }]); + const node = getRowTextNode(container, 0); + + expect( + extractSelectedLogText({ + container, + getRowText: () => "", + selection: selectBetween({ end: node, endOffset: 5, start: node, startOffset: 0 }), + }), + ).toBeUndefined(); + }); + + it("returns undefined when every selected row is mounted", () => { + const container = buildLogContainer([ + { index: 0, text: "line 0" }, + { index: 1, text: "line 1" }, + { index: 2, text: "line 2" }, + ]); + + expect( + extractSelectedLogText({ + container, + getRowText: () => "", + selection: selectBetween({ + end: getRowTextNode(container, 2), + endOffset: 3, + start: getRowTextNode(container, 0), + startOffset: 0, + }), + }), + ).toBeUndefined(); + }); + + it("rebuilds unmounted middle rows from log data, keeping partial boundary rows", () => { + const container = buildLogContainer([ + { index: 10, text: "hello world" }, + { index: 11, text: "mounted middle" }, + { index: 40, text: "foo bar" }, + ]); + const selection = selectBetween({ + end: getRowTextNode(container, 40), + endOffset: 3, + start: getRowTextNode(container, 10), + startOffset: 6, + }); + + const text = extractSelectedLogText({ + container, + getRowText: (index) => `line ${index}`, + selection, + }); + + const middleLines = Array.from({ length: 29 }, (_, offset) => `line ${offset + 11}`); + + expect(text).toBe(["world", ...middleLines, "foo"].join("\n")); + }); + + it("excludes copy-excluded line-number elements from boundary rows", () => { + const container = document.createElement("div"); + const buildRow = (index: number, messageText: string) => { + const row = document.createElement("div"); + + row.setAttribute("data-index", String(index)); + + const lineNumberLink = document.createElement("a"); + + lineNumberLink.setAttribute("data-copy-exclude", ""); + lineNumberLink.textContent = String(index); + + const message = document.createElement("span"); + + message.textContent = messageText; + row.append(lineNumberLink, message); + container.append(row); + + return message; + }; + const firstMessage = buildRow(10, "hello world"); + + buildRow(11, "mounted middle"); + + const lastMessage = buildRow(40, "foo bar"); + + document.body.append(container); + + const selection = selectBetween({ + end: lastMessage.firstChild as Node, + endOffset: 3, + start: firstMessage.firstChild as Node, + startOffset: 6, + }); + + const text = extractSelectedLogText({ + container, + getRowText: (index) => `line ${index}`, + selection, + }); + + const middleLines = Array.from({ length: 29 }, (_, offset) => `line ${offset + 11}`); + + expect(text).toBe(["world", ...middleLines, "foo"].join("\n")); + }); + + it("returns undefined for multi-range selections", () => { + const container = buildLogContainer([ + { index: 0, text: "line 0" }, + { index: 5, text: "line 5" }, + ]); + const range = document.createRange(); + + range.setStart(getRowTextNode(container, 0), 0); + range.setEnd(getRowTextNode(container, 5), 3); + + const selection = { getRangeAt: () => range, isCollapsed: false, rangeCount: 2 } as unknown as Selection; + + expect(extractSelectedLogText({ container, getRowText: () => "", selection })).toBeUndefined(); + }); + + it("returns undefined when the selection cannot be mapped to rows", () => { + const container = buildLogContainer([ + { index: 0, text: "line 0" }, + { index: 5, text: "line 5" }, + ]); + const range = document.createRange(); + + range.setStart(container, 0); + range.setEnd(getRowTextNode(container, 5), 3); + + expect( + extractSelectedLogText({ container, getRowText: () => "", selection: makeSelection(range) }), + ).toBeUndefined(); + }); +}); + +describe("getEntryText", () => { + it("returns string elements directly (group headers)", () => { + expect(getEntryText({ element: "Pre Execute" })).toBe("Pre Execute"); + }); + + it("prefers getPlainText over innerText for rendered log lines", () => { + const entry = { + element: createElement("span", undefined, "jsx text"), + getPlainText: () => "canonical text", + }; + + expect(getEntryText(entry)).toBe("canonical text"); + }); + + it("falls back to innerText when getPlainText is absent", () => { + expect(getEntryText({ element: createElement("span", undefined, "jsx text") })).toBe("jsx text"); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.ts b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.ts new file mode 100644 index 0000000000000..2c27549d90bf6 --- /dev/null +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/logSelection.ts @@ -0,0 +1,271 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import innerText from "react-innertext"; + +import type { ParsedLogEntry } from "src/queries/useLogs"; + +type RowRange = { + end: number; + start: number; +}; + +/** + * Map a DOM node inside the virtualized log list to the `data-index` of the + * row containing it. + */ +export const getRowIndexForNode = (node: Node | null, container: HTMLElement): number | undefined => { + const element = node instanceof Element ? node : node?.parentElement; + const row = element?.closest("[data-index]"); + + if (!row || !container.contains(row)) { + return undefined; + } + const index = Number(row.getAttribute("data-index")); + + return Number.isInteger(index) ? index : undefined; +}; + +/** + * Row-index range covered by the current text selection, if both selection + * boundaries sit inside log rows of `container`. + */ +export const getSelectionRowRange = ( + selection: Selection | null, + container: HTMLElement, +): RowRange | undefined => { + if (!selection || selection.isCollapsed || selection.rangeCount === 0) { + return undefined; + } + const range = selection.getRangeAt(0); + const start = getRowIndexForNode(range.startContainer, container); + const end = getRowIndexForNode(range.endContainer, container); + + if (start === undefined || end === undefined) { + return undefined; + } + + return start <= end ? { end, start } : { end: start, start: end }; +}; + +/** + * Row indexes to pin so the virtualizer keeps selection-boundary rows + * mounted. Boundaries map independently (the drag focus may sit off the + * rows) and a collapsed caret pins too, for shift-click extension. + */ +export const getSelectionPinnedRows = ( + selection: Selection | null, + container: HTMLElement, +): Array => { + if (!selection || selection.rangeCount === 0) { + return []; + } + const range = selection.getRangeAt(0); + + return [ + getRowIndexForNode(range.startContainer, container), + getRowIndexForNode(range.endContainer, container), + ].filter((index): index is number => index !== undefined); +}; + +type BottomDragClampOptions = { + container: HTMLElement; + pointerY: number; + selection: Selection; +}; + +type BottomDragBoundary = { + lastRow: Element; + y: number; +}; + +/** + * The first Y coordinate where a downward drag leaves selectable log text. + * Mid-scroll this is the scrollport edge; at the end it is the last mounted + * row edge, before any trailing space or container padding. + */ +export const getBottomDragBoundary = (container: HTMLElement): BottomDragBoundary | undefined => { + const rows = container.querySelectorAll("[data-index]"); + const lastRow = rows[rows.length - 1]; + + if (lastRow === undefined) { + return undefined; + } + + return { + lastRow, + y: Math.min(container.getBoundingClientRect().bottom, lastRow.getBoundingClientRect().bottom), + }; +}; + +/** + * Re-extend a downward drag selection to the last mounted row. When all log + * rows are absolutely positioned, Chrome can resolve a hit-test below the + * scrollport's text to the block start, reversing a downward selection. + */ +export const getBottomDragClampTarget = ({ + container, + pointerY, + selection, +}: BottomDragClampOptions): { node: Node; offset: number } | undefined => { + if (selection.rangeCount === 0) { + return undefined; + } + const anchorRow = getRowIndexForNode(selection.anchorNode, container); + + if (anchorRow === undefined) { + return undefined; + } + const boundary = getBottomDragBoundary(container); + + if (boundary === undefined || pointerY < boundary.y) { + return undefined; + } + const { lastRow } = boundary; + const offset = lastRow.childNodes.length; + + if (selection.focusNode === lastRow && selection.focusOffset === offset) { + return undefined; + } + + return { node: lastRow, offset }; +}; + +/** + * Merge selection-pinned row indexes into the virtualizer's default render + * range. Rows holding selection boundaries must stay mounted while the user + * scrolls — unmounting a boundary node collapses the browser selection. + */ +export const mergePinnedIndexes = ( + defaultIndexes: Array, + pinnedIndexes: Array, + count: number, +): Array => { + const validPins = pinnedIndexes.filter((index) => index >= 0 && index < count); + + if (validPins.length === 0) { + return defaultIndexes; + } + + return [...new Set([...validPins, ...defaultIndexes])].sort((first, second) => first - second); +}; + +/** + * Canonical plain text of a parsed log entry for clipboard rebuilding: + * group headers are plain strings, log lines render through the download + * text pipeline, and the innerText fallback covers synthetic entries such + * as the TI-context preamble. + */ +export const getEntryText = (entry: ParsedLogEntry): string => { + if (typeof entry.element === "string") { + return entry.element; + } + if (entry.getPlainText) { + return entry.getPlainText(); + } + + return entry.element ? innerText(entry.element) : ""; +}; + +/** + * Range text with copy-excluded elements (the line-number links) removed. + * Native copy drops them via `user-select: none`, but programmatic + * `Range.toString()` ignores CSS, so strip them explicitly. + */ +const getRangeText = (range: Range): string => { + const fragment = range.cloneContents(); + + for (const element of fragment.querySelectorAll("[data-copy-exclude]")) { + element.remove(); + } + + return fragment.textContent; +}; + +type ExtractSelectedLogTextOptions = { + container: HTMLElement; + getRowText: (index: number) => string; + selection: Selection; +}; + +/** + * Rebuild the text of a multi-row selection from log data. Native copy + * serializes the DOM, so it silently drops selected rows the virtualizer has + * unmounted. Returns undefined when native copy is already exact (single row + * or every selected row mounted) or when the selection cannot be mapped to + * log rows. + */ +export const extractSelectedLogText = ({ + container, + getRowText, + selection, +}: ExtractSelectedLogTextOptions): string | undefined => { + // Firefox multi-range selections: rebuilding only range 0 would clobber the rest. + if (selection.rangeCount !== 1) { + return undefined; + } + const rowRange = getSelectionRowRange(selection, container); + + if (!rowRange || rowRange.start === rowRange.end) { + return undefined; + } + + const mountedIndexes = new Set( + [...container.querySelectorAll("[data-index]")].map((row) => Number(row.getAttribute("data-index"))), + ); + + let hasUnmountedRow = false; + + for (let index = rowRange.start + 1; index < rowRange.end; index += 1) { + if (!mountedIndexes.has(index)) { + hasUnmountedRow = true; + break; + } + } + + if (!hasUnmountedRow) { + return undefined; + } + + const firstRow = container.querySelector(`[data-index="${rowRange.start}"]`); + const lastRow = container.querySelector(`[data-index="${rowRange.end}"]`); + + if (!firstRow || !lastRow) { + return undefined; + } + + const range = selection.getRangeAt(0); + const firstPartial = document.createRange(); + + firstPartial.selectNodeContents(firstRow); + firstPartial.setStart(range.startContainer, range.startOffset); + + const lastPartial = document.createRange(); + + lastPartial.selectNodeContents(lastRow); + lastPartial.setEnd(range.endContainer, range.endOffset); + + const lines = [getRangeText(firstPartial)]; + + for (let index = rowRange.start + 1; index < rowRange.end; index += 1) { + lines.push(getRowText(index)); + } + lines.push(getRangeText(lastPartial)); + + return lines.join("\n"); +}; diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.test.ts b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.test.ts index 004f9a2b6e0ce..6bb68d7192a90 100644 --- a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.test.ts +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.test.ts @@ -19,7 +19,7 @@ import type { TFunction } from "i18next"; import { describe, expect, it } from "vitest"; -import { getDownloadText, getHighlightColor, splitBySearchQuery } from "./utils"; +import { getDownloadText, getHighlightColor, isSelectionWithin, splitBySearchQuery } from "./utils"; const translate = ((key: string) => key) as unknown as TFunction; @@ -240,3 +240,79 @@ describe("splitBySearchQuery", () => { expect(splitBySearchQuery("error", "error")).toEqual([{ highlight: true, text: "error" }]); }); }); + +const makeSelectionForContainer = (options: { + anchor: Node | null; + collapsed?: boolean; + focus?: Node | null; + rangeCount?: number; +}): Selection => + ({ + anchorNode: options.anchor, + focusNode: options.focus ?? options.anchor, + isCollapsed: options.collapsed ?? false, + rangeCount: options.rangeCount ?? 1, + }) as unknown as Selection; + +const buildSelectionContainer = () => { + const container = document.createElement("div"); + const inside = document.createElement("span"); + + inside.textContent = "log line"; + container.append(inside); + + return { container, inside: inside.firstChild as Node }; +}; + +describe("isSelectionWithin", () => { + it("returns true when a non-collapsed selection sits inside the container", () => { + const { container, inside } = buildSelectionContainer(); + + expect(isSelectionWithin(makeSelectionForContainer({ anchor: inside }), container)).toBe(true); + }); + + it("returns true when only one boundary is inside the container", () => { + const { container, inside } = buildSelectionContainer(); + const outside = document.createElement("div"); + + outside.textContent = "elsewhere"; + + expect( + isSelectionWithin(makeSelectionForContainer({ anchor: outside.firstChild, focus: inside }), container), + ).toBe(true); + }); + + it("returns false for a collapsed selection (a plain click)", () => { + const { container, inside } = buildSelectionContainer(); + + expect(isSelectionWithin(makeSelectionForContainer({ anchor: inside, collapsed: true }), container)).toBe( + false, + ); + }); + + it("returns false when the selection is entirely outside the container", () => { + const { container } = buildSelectionContainer(); + const outside = document.createElement("div"); + + outside.textContent = "elsewhere"; + + expect(isSelectionWithin(makeSelectionForContainer({ anchor: outside.firstChild }), container)).toBe( + false, + ); + }); + + it("returns false for a null selection or null container", () => { + const { container, inside } = buildSelectionContainer(); + + expect(isSelectionWithin(null, container)).toBe(false); + expect(isSelectionWithin(makeSelectionForContainer({ anchor: inside }), null)).toBe(false); + }); + + it("returns false when there is no range", () => { + const { container, inside } = buildSelectionContainer(); + + expect(isSelectionWithin(makeSelectionForContainer({ anchor: inside, rangeCount: 0 }), container)).toBe( + false, + ); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.ts b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.ts index 248a89de2a589..5ff63bb551199 100644 --- a/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.ts +++ b/airflow-core/src/airflow/ui/src/pages/TaskInstance/Logs/utils.ts @@ -183,3 +183,16 @@ export const scrollToBottom = ({ element, virtualizer }: ScrollToBottomOptions): virtualizer.scrollToOffset(offset); element.scrollTop = offset; }; + +/** + * Whether a non-empty text selection currently sits inside `container`. + * Used to pause following new log lines while the user is selecting text, so + * auto-scrolling does not move the text out from under the cursor. + */ +export const isSelectionWithin = (selection: Selection | null, container: HTMLElement | null): boolean => { + if (!selection || selection.isCollapsed || selection.rangeCount === 0 || !container) { + return false; + } + + return container.contains(selection.anchorNode) || container.contains(selection.focusNode); +}; diff --git a/airflow-core/src/airflow/ui/src/queries/useLogs.test.ts b/airflow-core/src/airflow/ui/src/queries/useLogs.test.ts new file mode 100644 index 0000000000000..2f56c48e60849 --- /dev/null +++ b/airflow-core/src/airflow/ui/src/queries/useLogs.test.ts @@ -0,0 +1,58 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +import type { TFunction } from "i18next"; +import { describe, expect, it } from "vitest"; + +import type { StructuredLogMessage } from "openapi/requests/types.gen"; + +import { getLogLineText } from "./useLogs"; + +const translate = ((key: string) => key) as unknown as TFunction; + +describe("getLogLineText", () => { + it("renders a structured line with timestamp and level, stripping ANSI codes", () => { + const logMessage = { + event: "\u001B[31mfailed\u001B[0m to run", + level: "error", + timestamp: "2026-01-01T00:00:00Z", + } as StructuredLogMessage; + + expect(getLogLineText({ logMessage, showSource: false, showTimestamp: true, translate })).toBe( + "[2026-01-01T00:00:00Z] ERROR - failed to run", + ); + }); + + it("omits the timestamp when showTimestamp is false", () => { + const logMessage = { + event: "task done", + level: "info", + timestamp: "2026-01-01T00:00:00Z", + } as StructuredLogMessage; + + expect(getLogLineText({ logMessage, showSource: false, showTimestamp: false, translate })).toBe( + "INFO - task done", + ); + }); + + it("strips ANSI codes from plain string lines", () => { + expect(getLogLineText({ logMessage: "plain \u001B[32mok\u001B[0m line", translate })).toBe( + "plain ok line", + ); + }); +}); diff --git a/airflow-core/src/airflow/ui/src/queries/useLogs.tsx b/airflow-core/src/airflow/ui/src/queries/useLogs.tsx index 52f54e9263a7f..84af1d47be42f 100644 --- a/airflow-core/src/airflow/ui/src/queries/useLogs.tsx +++ b/airflow-core/src/airflow/ui/src/queries/useLogs.tsx @@ -17,6 +17,7 @@ * under the License. */ import type { UseQueryOptions } from "@tanstack/react-query"; +import Anser from "anser"; import dayjs from "dayjs"; import type { TFunction } from "i18next"; import type { JSX } from "react"; @@ -24,7 +25,11 @@ import { useTranslation } from "react-i18next"; import innerText from "react-innertext"; import { useTaskInstanceServiceGetLog } from "openapi/queries"; -import type { TaskInstanceResponse, TaskInstancesLogResponse } from "openapi/requests/types.gen"; +import type { + StructuredLogMessage, + TaskInstanceResponse, + TaskInstancesLogResponse, +} from "openapi/requests/types.gen"; import { extractTIContext, renderStructuredLog, @@ -36,10 +41,47 @@ import { parseStreamingLogContent } from "src/utils/logs"; export type ParsedLogEntry = { element: JSX.Element | string | undefined; + getPlainText?: () => string; group?: { id: number; level: number; parentId?: number; type: "header" | "line" }; lineNumber?: number; }; +type GetLogLineTextOptions = { + logLevelFilters?: Array; + logMessage: string | StructuredLogMessage; + showSource?: boolean; + showTimestamp?: boolean; + sourceFilters?: Array; + translate: TFunction; +}; + +/** + * Plain-text rendering of a single log line — same pipeline as the log + * download, with ANSI escape codes stripped. Used to rebuild copied text for + * selected rows the virtualizer has unmounted. + */ +export const getLogLineText = ({ + logLevelFilters, + logMessage, + showSource, + showTimestamp, + sourceFilters, + translate, +}: GetLogLineTextOptions): string => + Anser.ansiToText( + renderStructuredLog({ + index: 0, + logLevelFilters, + logLink: "", + logMessage, + renderingMode: "text", + showSource, + showTimestamp, + sourceFilters, + translate, + }), + ); + type Props = { accept?: "*/*" | "application/json" | "application/x-ndjson"; dagId: string; @@ -117,6 +159,7 @@ const parseLogs = ({ translate, }), lineNumber: lineNumbers[index], + logMessage: datum, }; }) .filter(({ element }) => element !== ""); @@ -136,8 +179,10 @@ const parseLogs = ({ const result: Array = []; let nextGroupId = 0; - parsedLines.forEach(({ element, lineNumber }) => { + parsedLines.forEach(({ element, lineNumber, logMessage }) => { const text = innerText(element); + const getPlainText = () => + getLogLineText({ logLevelFilters, logMessage, showSource, showTimestamp, sourceFilters, translate }); if (text.includes("::group::")) { const groupName = text.split("::group::")[1] as string; @@ -167,11 +212,12 @@ const parseLogs = ({ if (groupStack.length > 0 && currentGroup) { result.push({ element, + getPlainText, group: { id: currentGroup.id, level: currentGroup.level, type: "line" }, lineNumber, }); } else { - result.push({ element, lineNumber }); + result.push({ element, getPlainText, lineNumber }); } }); diff --git a/airflow-core/src/airflow/ui/src/utils/useFiltersHandler.ts b/airflow-core/src/airflow/ui/src/utils/useFiltersHandler.ts index 9349249c0783f..be6081ac1b526 100644 --- a/airflow-core/src/airflow/ui/src/utils/useFiltersHandler.ts +++ b/airflow-core/src/airflow/ui/src/utils/useFiltersHandler.ts @@ -59,6 +59,7 @@ export type FilterableSearchParamsKeys = | SearchParamsKeys.ASSET_EVENT_DATE_RANGE | SearchParamsKeys.BODY_SEARCH | SearchParamsKeys.BUNDLE_VERSION + | SearchParamsKeys.COMPLETED_AT_RANGE | SearchParamsKeys.CONF_CONTAINS | SearchParamsKeys.CONSUMING_ASSET_PATTERN | SearchParamsKeys.CREATED_AT_RANGE @@ -81,6 +82,8 @@ export type FilterableSearchParamsKeys = | SearchParamsKeys.LAST_ASSET_EVENT_TIMESTAMP_RANGE | SearchParamsKeys.LOGICAL_DATE_RANGE | SearchParamsKeys.MAP_INDEX + | SearchParamsKeys.MAX_ACTIVE_RUNS_GTE + | SearchParamsKeys.MAX_ACTIVE_RUNS_LTE | SearchParamsKeys.MISSED | SearchParamsKeys.NAME_PATTERN | SearchParamsKeys.OPERATOR_NAME_PATTERN @@ -88,6 +91,7 @@ export type FilterableSearchParamsKeys = | SearchParamsKeys.POOL_NAME_PATTERN | SearchParamsKeys.QUEUE_NAME_PATTERN | SearchParamsKeys.RENDERED_MAP_INDEX + | SearchParamsKeys.REPROCESS_BEHAVIOR | SearchParamsKeys.RESPONDED_BY_USER_NAME | SearchParamsKeys.RESPONSE_RECEIVED | SearchParamsKeys.RUN_AFTER_RANGE diff --git a/airflow-core/src/airflow/ui/src/vite-env.d.ts b/airflow-core/src/airflow/ui/src/vite-env.d.ts index e520ef77aa626..0efa1e9c8b0c0 100644 --- a/airflow-core/src/airflow/ui/src/vite-env.d.ts +++ b/airflow-core/src/airflow/ui/src/vite-env.d.ts @@ -25,14 +25,9 @@ interface ImportMeta { readonly env: ImportMetaEnv; } -// monaco-editor ships .d.ts only for `editor.api`; contribution side-effect imports have -// no typings of their own. -declare module "monaco-editor/esm/vs/editor/contrib/folding/browser/folding"; -declare module "monaco-editor/esm/vs/base/browser/ui/codicons/codiconStyles"; - -// The Python basic-language module exports its Monarch grammar (`conf` / `language`) +// The Python language definition exports its Monarch grammar (`conf` / `language`) // but ships no `.d.ts` of its own. -declare module "monaco-editor/esm/vs/basic-languages/python/python.js" { +declare module "monaco-editor/languages/definitions/python/python" { // `import(...)` type syntax is required here: a top-level `import type` would turn this // ambient declaration file into a module and break the `ImportMeta` augmentation above. /* eslint-disable @typescript-eslint/consistent-type-imports */ diff --git a/airflow-core/src/airflow/utils/db.py b/airflow-core/src/airflow/utils/db.py index 615ecb66c6799..7dcb15c742367 100644 --- a/airflow-core/src/airflow/utils/db.py +++ b/airflow-core/src/airflow/utils/db.py @@ -117,7 +117,7 @@ class MappedClassProtocol(Protocol): "3.1.8": "509b94a1042d", "3.2.0": "1d6611b6ab7c", "3.3.0": "d2f4e1b3c5a7", - "3.4.0": "7a98f1b7dbd3", + "3.4.0": "3c525f44bea8", } # Prefix used to identify tables holding data moved during migration. diff --git a/airflow-core/src/airflow/utils/event_scheduler.py b/airflow-core/src/airflow/utils/event_scheduler.py index 88999ec69372f..2d5dfa683ae80 100644 --- a/airflow-core/src/airflow/utils/event_scheduler.py +++ b/airflow-core/src/airflow/utils/event_scheduler.py @@ -32,12 +32,32 @@ def call_regular_interval( action: Callable, arguments=(), kwargs=None, + non_fatal: bool = False, ): - """Call a function at (roughly) a given interval.""" + """ + Call a function at (roughly) a given interval. + + :param non_fatal: If True, an exception raised by ``action`` is logged and + swallowed instead of propagating, so a single bad cycle can't kill + whatever is driving this scheduler. The next cycle is still scheduled either + way. Defaults to False (propagate), preserving prior behavior for callers + that rely on the exception surfacing. + """ def repeat(*args, **kwargs): self.log.debug("Calling %s", action) - action(*args, **kwargs) + if non_fatal: + try: + action(*args, **kwargs) + except Exception as e: + self.log.warning( + "Failed to run periodic action %s due to %s; will retry on the next cycle", + getattr(action, "__name__", action), + e, + exc_info=True, + ) + else: + action(*args, **kwargs) # This is not perfect. If we want a timer every 60s, but action # takes 10s to run, this will run it every 70s. # Good enough for now diff --git a/airflow-core/tests/unit/always/test_secrets.py b/airflow-core/tests/unit/always/test_secrets.py index 0c2749ca9d1fd..ff9aebb8e754f 100644 --- a/airflow-core/tests/unit/always/test_secrets.py +++ b/airflow-core/tests/unit/always/test_secrets.py @@ -122,6 +122,7 @@ def test_backend_fallback_to_env_var(self, mock_get_connection): assert conn.get_uri() == "mysql://airflow:airflow@host:5432/airflow" @pytest.mark.db_test + @conf_vars({("core", "multi_team"): "True"}) @mock.patch.dict( "os.environ", { @@ -218,6 +219,7 @@ def test_backend_variable_order(self, mock_secret_get, mock_meta_get): mock_secret_get.return_value = "a_secret_value" assert Variable.get(key="not_myvar") == "a_secret_value" + @conf_vars({("core", "multi_team"): "True"}) @mock.patch.dict( "os.environ", { diff --git a/airflow-core/tests/unit/always/test_secrets_environment_variables.py b/airflow-core/tests/unit/always/test_secrets_environment_variables.py index 18fb5d2444efd..200b545bf03a4 100644 --- a/airflow-core/tests/unit/always/test_secrets_environment_variables.py +++ b/airflow-core/tests/unit/always/test_secrets_environment_variables.py @@ -21,10 +21,13 @@ from airflow.secrets.environment_variables import ( CONN_ENV_PREFIX, + TEAM_SEP, VAR_ENV_PREFIX, EnvironmentVariablesBackend, ) +from tests_common.test_utils.config import conf_vars + # A team specific secret is stored as ``____``. Team names may contain # underscores (they are validated against ``^[a-zA-Z0-9_-]{3,50}$``), so both shapes are exercised. TEAM_NAMES = ["team_a", "teama"] @@ -56,6 +59,11 @@ def lookup(env_prefix: str, method: str, secret_id: str, team_name: str | None) class TestEnvironmentVariablesBackendTeamScope: """A team specific secret must only be resolvable for the team it is stored for.""" + @pytest.fixture(autouse=True) + def _multi_team_enabled(self): + with conf_vars({("core", "multi_team"): "True"}): + yield + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) @pytest.mark.parametrize("team_name", TEAM_NAMES) def test_team_scoped_secret_is_not_resolved_without_a_team_scope( @@ -136,9 +144,64 @@ def test_team_agnostic_secret_is_resolved_for_any_team_scope( assert lookup(env_prefix, method, SECRET_ID, team_name) == GLOBAL_VALUE + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) + def test_a_bare_id_containing_the_separator_is_not_resolved(self, monkeypatch, env_prefix, method): + """The scoped lookup can build another team's variable name from a *bare* id. + + Caller in ``team_a`` supplies ``prod___dbconn``. The team scoped lookup builds + ``_TEAM_A___PROD___DBCONN`` -- byte-identical to what team ``team_a___prod`` + builds for its own id ``dbconn``. That lookup **hits**, so refusing only the team + agnostic fall-through never sees this: the fall-through is never reached. + """ + target_team, target_id = "team_a___prod", "dbconn" + monkeypatch.setenv(team_env_var(env_prefix, target_team, target_id), TEAM_VALUE) + + assert lookup(env_prefix, method, f"prod{TEAM_SEP}{target_id}", "team_a") is None + + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) + @pytest.mark.parametrize("team_name", [None, *TEAM_NAMES]) + def test_an_id_containing_the_separator_is_refused_in_every_scope( + self, monkeypatch, env_prefix, method, team_name + ): + """One rule, applied the same way with or without a team scope. + + An id carrying the separator cannot be resolved unambiguously in either direction, so + there is no scope in which it is safe to look up. + """ + monkeypatch.setenv(env_prefix + f"A{TEAM_SEP}B".upper(), GLOBAL_VALUE) + + assert lookup(env_prefix, method, f"a{TEAM_SEP}b", team_name) is None + + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) + @pytest.mark.parametrize("team_name", TEAM_NAMES) + def test_the_owning_team_cannot_reach_a_separator_bearing_id_either( + self, monkeypatch, env_prefix, method, team_name + ): + """Documents the cost of the rule, so a future change does not reintroduce the hole. + + A team's *own* secret whose id contains the separator is unreachable too. That is + deliberate: the string it builds is the same one another team's name could build, and + nothing in it says which reading is intended. + """ + monkeypatch.setenv(team_env_var(env_prefix, team_name, f"prod{TEAM_SEP}dbconn"), TEAM_VALUE) + + assert lookup(env_prefix, method, f"prod{TEAM_SEP}dbconn", team_name) is None + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) @pytest.mark.parametrize("team_name", [None, *TEAM_NAMES]) def test_unset_secret_is_not_resolved(self, monkeypatch, env_prefix, method, team_name): monkeypatch.delenv(env_prefix + SECRET_ID.upper(), raising=False) assert lookup(env_prefix, method, SECRET_ID, team_name) is None + + +class TestEnvironmentVariablesBackendMultiTeamDisabled: + """No team scoped variable can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + + @pytest.mark.parametrize(("env_prefix", "method"), LOOKUPS) + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self, monkeypatch, env_prefix, method): + secret_id = f"prod{TEAM_SEP}{SECRET_ID}" + monkeypatch.setenv(env_prefix + secret_id.upper(), GLOBAL_VALUE) + + assert lookup(env_prefix, method, secret_id, None) == GLOBAL_VALUE diff --git a/airflow-core/tests/unit/api_fastapi/common/test_cursors.py b/airflow-core/tests/unit/api_fastapi/common/test_cursors.py index f101efebc9d8b..bb72908d28aab 100644 --- a/airflow-core/tests/unit/api_fastapi/common/test_cursors.py +++ b/airflow-core/tests/unit/api_fastapi/common/test_cursors.py @@ -307,3 +307,56 @@ def test_nullable_column_without_nulls_unaffected(self): session.close() assert collected == [1, 2, 3, 4, 5] + + @staticmethod + def _walk_backward(session, model, sort, page_size): + """Walk pages from the end to the start: ``_walk_forward`` with the direction flipped. + + Backward paging is the documented mirror of forward paging (see ``apply_cursor_filter``): + reverse the ORDER BY and select rows *before* the cursor. Rows come back in reverse of + the forward order, so the returned list reversed should equal ``_walk_forward``. + """ + collected: list[int] = [] + token = None + for _ in range(50): # guard against an infinite paging loop + stmt = sort.to_orm(select(model), reversed=True).limit(page_size) + if token is not None: + stmt = apply_cursor_filter(stmt, token, sort, "sqlite", is_backward=True) + rows = list(session.scalars(stmt)) + if not rows: + break + collected.extend(r.id for r in rows) + token = encode_cursor(rows[-1], sort) + return collected + + @pytest.mark.parametrize( + ("rows", "order_by", "page_size"), + [ + pytest.param([(1, "0"), (2, "0")], ["val"], 1, id="equal-values-tie-broken-by-pk"), + pytest.param([(1, "5"), (2, None)], ["-val"], 1, id="null-sorts-last-descending"), + pytest.param( + [(1, None), (2, None), (3, "a"), (4, "b"), (5, "c")], ["val"], 2, id="nulls-first-asc" + ), + pytest.param( + [(1, None), (2, None), (3, "a"), (4, "b"), (5, "c")], ["-val"], 2, id="nulls-last-desc" + ), + ], + ) + def test_backward_pagination_mirrors_forward(self, rows, order_by, page_size): + """Paging backward yields the same rows as paging forward, only reversed. + + The backward path must exclude only the boundary row (the last sort key uses a strict + bound; earlier keys stay inclusive) and invert NULL placement; otherwise a page walked + back into is silently truncated. + """ + model, session = self._seed_session(rows) + try: + sort = SortParam(["val"], model) + sort.set_value(order_by) + forward = self._walk_forward(session, model, sort, page_size) + backward = self._walk_backward(session, model, sort, page_size) + finally: + session.close() + + assert list(reversed(backward)) == forward + assert sorted(forward) == sorted(row[0] for row in rows), "rows dropped" diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_assets.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_assets.py index fcf411628723a..462d1605af05a 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_assets.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_assets.py @@ -25,6 +25,7 @@ from sqlalchemy import delete, func, select, update from airflow._shared.timezones import timezone +from airflow.api_fastapi.auth.managers.base_auth_manager import BaseAuthManager from airflow.api_fastapi.auth.managers.models.resource_details import DagAccessEntity, DagDetails from airflow.models import DagModel from airflow.models.asset import ( @@ -1504,7 +1505,10 @@ class TestQueuedEventEndpoint(TestAssets): def _create_asset_dag_run_queues(self, dag_id, asset_id, session): session.execute(delete(AssetDagRunQueue)) session.flush() - adrq = AssetDagRunQueue(target_dag_id=dag_id, asset_id=asset_id) + event = AssetEvent(asset_id=asset_id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + adrq = AssetDagRunQueue(target_dag_id=dag_id, asset_id=asset_id, asset_event_id=event.id) session.add(adrq) session.commit() return adrq @@ -2151,6 +2155,33 @@ def test_should_respond_200_with_partition_date_for_partitioned_dag( assert dag_run.partition_key == "2025-06-01T00:00:00" assert dag_run.partition_date == timezone.datetime(2025, 6, 1) + @pytest.mark.parametrize("team_name", ["team_b", None]) + def test_authorizes_against_the_dags_team(self, test_client, session, team_name): + """The Dag is resolved from the asset, so its team must be resolved and passed too — see the + call site's comment for why an unresolved team asks about the wrong resource.""" + recorded = [] + + auth_manager = mock.Mock(spec=BaseAuthManager) + auth_manager.is_authorized_dag.side_effect = lambda **kw: recorded.append(kw) or True + + with ( + mock.patch( + "airflow.api_fastapi.core_api.routes.public.assets.get_auth_manager", + return_value=auth_manager, + ), + mock.patch.object( + DagModel, "get_team_name", return_value=team_name, autospec=True + ) as mock_get_team_name, + ): + test_client.post("/assets/1/materialize") + + assert len(recorded) == 1, "expected exactly one authorization check" + details = recorded[0]["details"] + assert details.id == self.DAG_ASSET1_ID + assert details.team_name == team_name + # resolved for the Dag the asset led to, not for some other Dag + mock_get_team_name.assert_called_once_with(self.DAG_ASSET1_ID, session=mock.ANY) + class TestGetAssetQueuedEvents(TestQueuedEventEndpoint): @pytest.mark.usefixtures("time_freezer") @@ -2198,10 +2229,10 @@ def test_should_respond_204(self, test_client, session, create_dummy_dag): (asset,) = self.create_assets(session=session, num=1) self._create_asset_dag_run_queues(dag_id, asset.id, session) - assert session.get(AssetDagRunQueue, (asset.id, dag_id)) is not None + assert session.scalars(select(AssetDagRunQueue)).all() response = test_client.delete(f"/assets/{asset.id}/queuedEvents") assert response.status_code == 204 - assert session.get(AssetDagRunQueue, (asset.id, dag_id)) is None + assert session.scalars(select(AssetDagRunQueue)).all() == [] check_last_log(session, dag_id=None, event="delete_asset_queued_events", logical_date=None) def test_should_respond_401(self, unauthenticated_test_client): diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py index cebd1cf12118e..8880b474cf93c 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_config.py @@ -22,6 +22,8 @@ import pytest +from airflow.configuration import conf + from tests_common.test_utils.config import conf_vars pytestmark = pytest.mark.db_test @@ -559,3 +561,65 @@ def test_get_config_value_masks_per_key_secrets_backend_kwarg(self, test_client) ) assert response.status_code == 200 assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_SENSITIVE_HIDDEN + + +TEAM_NAME = "team_a" +SECTION_TEAM_DATABASE = f"{TEAM_NAME}={SECTION_DATABASE}" +SECTION_TEAM_CORE = f"{TEAM_NAME}={SECTION_CORE}" +OPTION_VALUE_TEAM_SQL_ALCHEMY_CONN = "postgresql://team_a_user:team_a_password@team-a-db/airflow" +OPTION_VALUE_TEAM_PARALLELISM = "512" + + +class TestTeamScopedOptionMasking(TestConfigEndpoint): + """A team scoped override of a sensitive option lives in a ``[=
]`` section rather + than in the section the option is registered under. The single-option endpoint has to resolve + that spelling back to the registered option, so that it redacts a team scoped value exactly as + it redacts the base one.""" + + @pytest.fixture(autouse=True) + def setup_team_scoped(self) -> Generator[None, None, None]: + team_config = { + SECTION_TEAM_DATABASE: {OPTION_KEY_SQL_ALCHEMY_CONN: OPTION_VALUE_TEAM_SQL_ALCHEMY_CONN}, + SECTION_TEAM_CORE: {OPTION_KEY_PARALLELISM: OPTION_VALUE_TEAM_PARALLELISM}, + } + + # Anything that is not one of the team scoped options under test keeps being read from the + # real configuration - the endpoint reads [api] expose_config through the same object. + real_get = conf.get + + def _mock_has_option(section: str, option: str, **_) -> bool: + return option in team_config.get(section, {}) + + def _mock_get(section: str, option: str, *args, **kwargs) -> str: + if option in team_config.get(section, {}): + return team_config[section][option] + return real_get(section, option, *args, **kwargs) + + with ( + conf_vars(AIRFLOW_CONFIG_ENABLE_EXPOSE_CONFIG), + patch( + "airflow.api_fastapi.core_api.routes.public.config.conf.has_option", + new=_mock_has_option, + ), + patch( + "airflow.api_fastapi.core_api.routes.public.config.conf.get", + new=_mock_get, + ), + ): + yield + + def test_get_config_value_masks_team_scoped_sensitive_option(self, test_client): + response = test_client.get( + f"/config/section/{SECTION_TEAM_DATABASE}/option/{OPTION_KEY_SQL_ALCHEMY_CONN}", + headers=HEADERS_JSON, + ) + assert response.status_code == 200 + assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_SENSITIVE_HIDDEN + + def test_get_config_value_keeps_team_scoped_non_sensitive_option(self, test_client): + response = test_client.get( + f"/config/section/{SECTION_TEAM_CORE}/option/{OPTION_KEY_PARALLELISM}", + headers=HEADERS_JSON, + ) + assert response.status_code == 200 + assert response.json()["sections"][0]["options"][0]["value"] == OPTION_VALUE_TEAM_PARALLELISM diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py index 97a54d1ef572f..0a14f69bcbeac 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py @@ -4394,6 +4394,32 @@ def test_should_respond_403_when_user_lacks_xcom_permission(self, test_client): user=mock.ANY, ) + @pytest.mark.parametrize("team_name", ["team_b", None]) + def test_authorizes_xcom_against_the_dags_team(self, test_client, team_name): + """The XCom check must carry the Dag's team, matching what the route dependency above already + resolves — see the call site's comment for why.""" + with ( + mock.patch( + "airflow.api_fastapi.core_api.routes.public.dag_run.get_auth_manager", + autospec=True, + ) as mock_get_auth_manager, + mock.patch.object(DagModel, "get_team_name", return_value=team_name, autospec=True), + ): + mock_get_auth_manager.return_value.is_authorized_dag.return_value = True + + response = test_client.get( + f"/dags/{DAG1_ID}/dagRuns/{DAG1_RUN1_ID}/wait", + params={"interval": "1", "result": "task_1"}, + ) + + assert response.status_code == 200 + mock_get_auth_manager.return_value.is_authorized_dag.assert_called_once_with( + method="GET", + access_entity=DagAccessEntity.XCOM, + details=DagDetails(id=DAG1_ID, team_name=team_name), + user=mock.ANY, + ) + def test_should_respond_200_without_result_when_user_lacks_xcom_permission(self, test_client): """Waiting without result parameter should not require XCom permissions.""" with mock.patch( diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py index 05735154c5f6c..2427729035fea 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py @@ -137,9 +137,15 @@ def test_should_set_last_update_only_for_queued_and_hide_flag(self, test_client, ) } # Queue and add an event only for A - session.add(AssetDagRunQueue(asset_id=assets["s3://bucket/A"].id, target_dag_id="two_assets_equal")) + event = AssetEvent(asset_id=assets["s3://bucket/A"].id, timestamp=dr.logical_date or pendulum.now()) + session.add(event) + session.flush() session.add( - AssetEvent(asset_id=assets["s3://bucket/A"].id, timestamp=dr.logical_date or pendulum.now()) + AssetDagRunQueue( + asset_id=assets["s3://bucket/A"].id, + target_dag_id="two_assets_equal", + asset_event_id=event.id, + ) ) session.commit() @@ -210,12 +216,16 @@ def test_last_update_respects_latest_run_filter(self, test_client, dag_maker, se dag_maker.sync_dagbag_to_db() asset = session.scalars(select(AssetModel).where(AssetModel.uri == "s3://bucket/F")).one() - session.add(AssetDagRunQueue(asset_id=asset.id, target_dag_id="filter_run")) - # event before latest_run should be ignored ts_base = dr.logical_date or pendulum.now() - session.add(AssetEvent(asset_id=asset.id, timestamp=ts_base.subtract(minutes=10))) + # event before latest_run should be ignored + event_before = AssetEvent(asset_id=asset.id, timestamp=ts_base.subtract(minutes=10)) # event after latest_run counts - session.add(AssetEvent(asset_id=asset.id, timestamp=ts_base.add(minutes=10))) + event_after = AssetEvent(asset_id=asset.id, timestamp=ts_base.add(minutes=10)) + session.add_all([event_before, event_after]) + session.flush() + session.add( + AssetDagRunQueue(asset_id=asset.id, target_dag_id="filter_run", asset_event_id=event_after.id) + ) session.commit() resp = test_client.get("/next_run_assets/filter_run") diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_backfills.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_backfills.py index ebb23efff7d4a..36e2d94a757a5 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_backfills.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_backfills.py @@ -16,13 +16,14 @@ # under the License. from __future__ import annotations +from datetime import timedelta from unittest import mock import pytest from airflow._shared.timezones import timezone from airflow.models import DagModel -from airflow.models.backfill import Backfill +from airflow.models.backfill import Backfill, ReprocessBehavior from airflow.utils.session import provide_session from tests_common.test_utils.asserts import assert_queries_count @@ -160,6 +161,88 @@ def test_should_response_200( "total_entries": total_entries, } + def test_list_backfill_by_start_end_date(self, test_client, session, testing_dag_bundle): + (dag,) = self._create_dag_models(count=1) + from_date = timezone.utcnow() + b1 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date) + other_date = from_date + timedelta(days=1) + b2 = Backfill(dag_id=dag.dag_id, from_date=other_date, to_date=other_date) + session.add_all([b1, b2]) + session.commit() + + response = test_client.get( + "/backfills", + params={ + "from_date_gte": from_datetime_to_zulu(other_date), + "to_date_lte": from_datetime_to_zulu(other_date), + }, + ) + assert response.status_code == 200 + assert [each["id"] for each in response.json()["backfills"]] == [b2.id] + + def test_list_backfill_by_completed_at(self, test_client, session, testing_dag_bundle): + (dag,) = self._create_dag_models(count=1) + from_date = timezone.utcnow() + b1 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date) + other_completed_at = timezone.utcnow() + timedelta(days=1) + b2 = Backfill( + dag_id=dag.dag_id, from_date=from_date, to_date=from_date, completed_at=other_completed_at + ) + session.add_all([b1, b2]) + session.commit() + + # b1's completed_at is NULL, so it never matches a bound + response = test_client.get( + "/backfills", params={"completed_at_gte": from_datetime_to_zulu(other_completed_at)} + ) + assert response.status_code == 200 + assert [each["id"] for each in response.json()["backfills"]] == [b2.id] + + def test_list_backfill_by_created_at(self, test_client, session, testing_dag_bundle): + (dag,) = self._create_dag_models(count=1) + from_date = timezone.utcnow() + b1 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date) + other_created_at = timezone.utcnow() - timedelta(days=1) + b2 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date, created_at=other_created_at) + session.add_all([b1, b2]) + session.commit() + + # b1's created_at defaults to ~now, b2's is 1 day in the past + response = test_client.get( + "/backfills", params={"created_at_lte": from_datetime_to_zulu(other_created_at)} + ) + assert response.status_code == 200 + assert [each["id"] for each in response.json()["backfills"]] == [b2.id] + + def test_list_backfill_by_reprocess_behavior(self, test_client, session, testing_dag_bundle): + (dag,) = self._create_dag_models(count=1) + from_date = timezone.utcnow() + b1 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date) + b2 = Backfill( + dag_id=dag.dag_id, + from_date=from_date, + to_date=from_date, + reprocess_behavior=ReprocessBehavior.COMPLETED, + ) + session.add_all([b1, b2]) + session.commit() + + response = test_client.get("/backfills", params={"reprocess_behavior": "completed"}) + assert response.status_code == 200 + assert [each["id"] for each in response.json()["backfills"]] == [b2.id] + + def test_list_backfill_by_max_active_runs(self, test_client, session, testing_dag_bundle): + (dag,) = self._create_dag_models(count=1) + from_date = timezone.utcnow() + b1 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date) # default max_active_runs=10 + b2 = Backfill(dag_id=dag.dag_id, from_date=from_date, to_date=from_date, max_active_runs=3) + session.add_all([b1, b2]) + session.commit() + + response = test_client.get("/backfills", params={"max_active_runs_lte": 5}) + assert response.status_code == 200 + assert [each["id"] for each in response.json()["backfills"]] == [b2.id] + def test_should_response_401(self, unauthenticated_test_client): response = unauthenticated_test_client.get("/backfills", params={}) assert response.status_code == 401 diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_dags.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_dags.py index 14c7c73f657e9..33f4a5441d542 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_dags.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_dags.py @@ -175,6 +175,24 @@ def test_dag_run_state_matches_any_run_not_only_latest(self, test_client, sessio assert any_state.status_code == 200 assert [dag["dag_id"] for dag in any_state.json()["dags"]] == [DAG1_ID] + @pytest.mark.usefixtures("configure_git_connection_for_dag_bundle") + def test_last_and_any_run_state_filters_combined(self, test_client, session): + # Regression: combining the last-run and any-run state filters must return the + # intersection, not raise. The any-run EXISTS subquery must not correlate to the + # DagRun the last-run filter joins into the outer query. + latest_run = session.scalar( + select(DagRun).where(DagRun.dag_id == DAG1_ID, DagRun.run_id == "run_id_5") + ) + latest_run.state = DagRunState.FAILED + session.commit() + + response = test_client.get( + "/dags", + params={"last_dag_run_state": "failed", "dag_run_state": "failed", "dag_ids": [DAG1_ID]}, + ) + assert response.status_code == 200 + assert [dag["dag_id"] for dag in response.json()["dags"]] == [DAG1_ID] + @pytest.fixture def setup_hitl_data(self, create_task_instance: TaskInstance, session: Session): """Setup HITL test data for parametrized tests.""" diff --git a/airflow-core/tests/unit/api_fastapi/core_api/test_security.py b/airflow-core/tests/unit/api_fastapi/core_api/test_security.py index 37750e30da959..3709d50d3580b 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/test_security.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/test_security.py @@ -20,12 +20,14 @@ from unittest.mock import AsyncMock, Mock, patch import pytest -from fastapi import HTTPException +from fastapi import HTTPException, Request from jwt import ExpiredSignatureError, InvalidTokenError +from sqlalchemy.orm import Session from airflow import settings from airflow.api_fastapi.app import create_app -from airflow.api_fastapi.auth.managers.base_auth_manager import COOKIE_NAME_JWT_TOKEN +from airflow.api_fastapi.auth.managers.base_auth_manager import COOKIE_NAME_JWT_TOKEN, BaseAuthManager +from airflow.api_fastapi.auth.managers.models.base_user import BaseUser from airflow.api_fastapi.auth.managers.models.resource_details import ( AccessView, ConnectionDetails, @@ -55,6 +57,7 @@ resolve_user_from_token, ) from airflow.models import Connection, Pool, Variable +from airflow.models.backfill import Backfill from airflow.models.dag import DagModel from airflow.models.dagbundle import DagBundleModel from airflow.models.team import Team @@ -367,7 +370,7 @@ async def test_requires_access_backfill_authorized_from_path( async def test_requires_access_backfill_authorized_from_body( self, mock_get_auth_manager, mock_get_team_name ): - """When backfill_id is missing or not int, dag_id can come from request body (POST backfill).""" + """With no backfill_id in the path, dag_id comes from the request body (POST backfill).""" auth_manager = Mock() auth_manager.is_authorized_dag.return_value = True mock_get_auth_manager.return_value = auth_manager @@ -455,6 +458,48 @@ async def test_requires_access_backfill_backfill_not_found_falls_back_to_body( user=user, ) + @pytest.mark.db_test + @pytest.mark.asyncio + @pytest.mark.parametrize("backfill_id", ["42", "42.0", "42.00"]) + @patch.object(DagModel, "get_team_name") + @patch("airflow.api_fastapi.core_api.security.get_auth_manager") + async def test_requires_access_backfill_authorizes_the_backfill_the_handler_will_act_on( + self, mock_get_auth_manager, mock_get_team_name, backfill_id + ): + """The dependency must resolve the same backfill the handler does, for every spelling. + + The endpoints declare ``backfill_id: NonNegativeInt``, and pydantic's lax mode coerces + ``"42.0"`` and ``"42.00"`` to ``42`` -- both are spellings the handler accepts and serves + against backfill 42. Parsing with ``int()`` here rejected them and left ``dag_id`` + unresolved, so the two disagreed about which Dag the request concerned. + """ + auth_manager = Mock(spec=BaseAuthManager) + auth_manager.is_authorized_dag.return_value = True + mock_get_auth_manager.return_value = auth_manager + mock_get_team_name.return_value = "team1" + + backfill = Mock(spec=Backfill) + backfill.dag_id = "backfill_dag" + session = Mock(spec=Session) + session.scalars.return_value.one_or_none.return_value = backfill + + request = Mock(spec=Request) + request.path_params = {"backfill_id": backfill_id} + request.query_params = {"dag_id": "some_other_dag"} + request.json = AsyncMock(return_value={"dag_id": "some_other_dag"}) + + user = Mock(spec=BaseUser) + + await requires_access_backfill("PUT")(request, user, session) + + # the backfill's own Dag, not the one supplied on the request + auth_manager.is_authorized_dag.assert_called_once_with( + method="PUT", + access_entity=DagAccessEntity.RUN, + details=DagDetails(id="backfill_dag", team_name="team1"), + user=user, + ) + @pytest.mark.db_test @pytest.mark.asyncio @patch.object(DagModel, "get_team_name") diff --git a/airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py b/airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py index 8a152bebe0d3f..bb3c0f7e5a785 100644 --- a/airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py +++ b/airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_task_instances.py @@ -66,6 +66,7 @@ clear_db_serialized_dags, clear_rendered_ti_fields, ) +from unit.listeners import asset_listener if TYPE_CHECKING: from airflow.sdk.api.client import Client @@ -1295,6 +1296,83 @@ def test_ti_update_state_to_success_with_asset_events( assert event[0].asset == AssetModel(name="my-task", uri="s3://bucket/my-task", extra={}) assert event[0].extra == expected_extra + def test_ti_update_state_to_success_runs_deferred_asset_listener_callbacks( + self, client, session, create_task_instance, listener_manager + ): + """The success endpoint runs the deferred asset listener callbacks after committing.""" + asset_listener.clear() + listener_manager(asset_listener) + + asset = AssetModel(id=1, name="my-task", uri="s3://bucket/my-task", group="asset", extra={}) + session.add_all([asset, AssetActive.for_asset(asset)]) + + ti = create_task_instance( + task_id="test_ti_update_state_to_success_runs_deferred_asset_listener_callbacks", + start_date=DEFAULT_START_DATE, + state=State.RUNNING, + ) + session.commit() + + response = client.patch( + f"/execution/task-instances/{ti.id}/state", + json={ + "state": "success", + "end_date": DEFAULT_END_DATE.isoformat(), + "task_outlets": [{"name": "my-task", "uri": "s3://bucket/my-task", "type": "Asset"}], + "outlet_events": [], + }, + ) + + assert response.status_code == 204 + + # Notifications are deferred during registration and run by the endpoint after the + # TI state is committed (and the task_instance row lock released). + assert len(asset_listener.changed) == 1 + assert asset_listener.changed[0].uri == "s3://bucket/my-task" + assert len(asset_listener.emitted) == 1 + + def test_ti_update_state_to_success_rolls_back_partial_asset_registration( + self, client, session, create_task_instance + ): + """A failure partway through asset registration rolls back the partial writes. + + The endpoint's exception handler marks the TI failed and commits; the explicit rollback + ensures an asset event flushed before the failure is not committed alongside it. + """ + asset = AssetModel(id=1, name="my-task", uri="s3://bucket/my-task", group="asset", extra={}) + session.add_all([asset, AssetActive.for_asset(asset)]) + + ti = create_task_instance( + task_id="test_ti_update_state_to_success_rolls_back_partial_asset_registration", + start_date=DEFAULT_START_DATE, + state=State.RUNNING, + ) + session.commit() + + def _partial_then_fail(ti, task_outlets, outlet_events, *, session): + # Simulate a half-way registration: an asset event is flushed, then registration + # fails before the endpoint commits. + session.add(AssetEvent(asset_id=asset.id)) + session.flush() + raise RuntimeError("boom partway through outlets") + + with mock.patch.object(TaskInstance, "register_asset_changes_in_db", side_effect=_partial_then_fail): + response = client.patch( + f"/execution/task-instances/{ti.id}/state", + json={ + "state": "success", + "end_date": DEFAULT_END_DATE.isoformat(), + "task_outlets": [{"name": "my-task", "uri": "s3://bucket/my-task", "type": "Asset"}], + "outlet_events": [], + }, + ) + + assert response.status_code == 204 + session.expire_all() + # The partially-written asset event was rolled back, and the TI is marked failed. + assert session.scalars(select(AssetEvent)).all() == [] + assert session.get(TaskInstance, ti.id).state == State.FAILED + @pytest.mark.parametrize( ("outlet_events", "expected_extra"), [ diff --git a/airflow-core/tests/unit/api_fastapi/logging/test_decorators.py b/airflow-core/tests/unit/api_fastapi/logging/test_decorators.py index 2adffbe454494..ea356b8703622 100644 --- a/airflow-core/tests/unit/api_fastapi/logging/test_decorators.py +++ b/airflow-core/tests/unit/api_fastapi/logging/test_decorators.py @@ -130,6 +130,156 @@ def test_value_without_key_is_still_masked(self): assert result == {"value": "***"} +class TestMaskBulkFields: + """The bulk endpoints nest their entities, and the masking has to reach them.""" + + def test_bulk_variable_values_are_masked(self): + result = _mask_variable_fields( + { + "actions": [ + { + "action": "create", + "entities": [ + {"key": "campaign_signing_material", "value": "VARVAL_LEAK_token"}, + {"key": "other", "val": "VARVAL_LEAK_alias"}, + ], + "action_on_existence": "overwrite", + } + ] + } + ) + assert result == { + "actions": [ + { + "action": "create", + "entities": [ + {"key": "campaign_signing_material", "value": "***"}, + {"key": "other", "val": "***"}, + ], + "action_on_existence": "overwrite", + } + ] + } + + def test_bulk_connection_extra_is_masked(self): + """``extra`` is the load-bearing case for connections. + + Key-name redaction already covered a nested ``password`` -- but only while + ``hide_sensitive_var_conn_fields`` is enabled, which is a deployment setting and is off + in this test environment. ``extra`` was never covered by it at all: the name is not a + recognised sensitive field, and the value is a JSON *string*, which ``redact`` returns + unchanged. Masking ``extra`` is structural here, so it does not depend on that setting. + """ + result = _mask_connection_fields( + { + "actions": [ + { + "action": "create", + "entities": [ + { + "connection_id": "c1", + "conn_type": "http", + "password": "CONN_LEAK_pw", + "extra": json.dumps({"token": "CONN_LEAK_token", "region": "eu"}), + } + ], + } + ] + } + ) + entity = result["actions"][0]["entities"][0] + assert entity["extra"] == {"token": "***", "region": "***"} + assert entity["connection_id"] == "c1" + # every value is gone, only the key names of ``extra`` remain + assert "CONN_LEAK_token" not in json.dumps(result) + + @pytest.mark.parametrize( + ("body", "masker"), + [ + ( + {"actions": [{"action": "create", "entities": [{"key": "k", "value": "LEAK_v"}]}]}, + _mask_variable_fields, + ), + ( + { + "actions": [ + { + "action": "update", + "entities": [{"connection_id": "c", "extra": json.dumps({"t": "LEAK_v"})}], + } + ] + }, + _mask_connection_fields, + ), + ], + ids=["variables", "connections"], + ) + def test_no_secret_survives_in_the_serialized_entry(self, body, masker): + """The audit entry is serialized whole, so assert on the serialization, not one field.""" + assert "LEAK_v" not in json.dumps(masker(body)) + + def test_multiple_actions_and_entities_are_all_masked(self): + result = _mask_variable_fields( + { + "actions": [ + { + "action": "create", + "entities": [{"key": "a", "value": "s1"}, {"key": "b", "value": "s2"}], + }, + {"action": "update", "entities": [{"key": "c", "value": "s3"}]}, + ] + } + ) + values = [e["value"] for a in result["actions"] for e in a["entities"]] + assert values == ["***", "***", "***"] + + def test_delete_by_key_entities_are_left_alone(self): + """``delete`` may list bare keys rather than entity objects; those carry no secret.""" + body = {"actions": [{"action": "delete", "entities": ["key_one", "key_two"]}]} + assert _mask_variable_fields(body) == body + + @pytest.mark.parametrize( + "body", + [ + {"key": "k", "value": "secret"}, + {"actions": "not-a-list"}, + {"actions": [{"action": "create"}]}, + {"actions": [{"action": "create", "entities": "not-a-list"}]}, + {"actions": ["not-a-dict"]}, + ], + ids=["flat-body", "actions-not-list", "no-entities", "entities-not-list", "action-not-dict"], + ) + def test_non_bulk_and_malformed_shapes_do_not_raise(self, body): + """The masker runs on request bodies before validation, so it must not add a failure mode.""" + _mask_variable_fields(body) + _mask_connection_fields(body) + + @pytest.mark.parametrize( + "extra", + [ + pytest.param({"token": "CONN_LEAK_token"}, id="already-decoded-object"), + pytest.param(["CONN_LEAK_token"], id="already-decoded-array"), + pytest.param(123, id="number"), + pytest.param(True, id="bool"), + ], + ) + def test_non_string_extra_does_not_raise_and_does_not_leak(self, extra): + """``extra`` reaches this before validation, so it need not be a string. + + ``json.loads`` raises ``TypeError`` rather than ``JSONDecodeError`` for a non-string, which + would otherwise escape the audit-log decorator. + """ + body = {"actions": [{"action": "create", "entities": [{"connection_id": "c1", "extra": extra}]}]} + + result = _mask_connection_fields(body) + + assert "CONN_LEAK_token" not in json.dumps(result) + assert result["actions"][0]["entities"][0]["connection_id"] == "c1" + + def test_flat_body_still_takes_the_single_entity_path(self): + assert _mask_variable_fields({"key": "k", "value": "secret"}) == {"key": "k", "value": "***"} + + class TestActionLoggingUserFields: """The audit Log records the raw identifier in ``owner`` and the human-friendly ``get_display_name()`` in ``owner_display_name``.""" diff --git a/airflow-core/tests/unit/assets/test_manager.py b/airflow-core/tests/unit/assets/test_manager.py index bc034cd751540..3d9d75ff12acb 100644 --- a/airflow-core/tests/unit/assets/test_manager.py +++ b/airflow-core/tests/unit/assets/test_manager.py @@ -25,8 +25,8 @@ from unittest import mock import pytest -from sqlalchemy import delete, func, select -from sqlalchemy.dialects import mysql +from sqlalchemy import func, select +from sqlalchemy.dialects import mysql, postgresql, sqlite from sqlalchemy.orm import Session from airflow import settings @@ -67,7 +67,7 @@ pytest.importorskip("pydantic", minversion="2.0.0") -@pytest.fixture +@pytest.fixture(autouse=True) def clear_assets(): from tests_common.test_utils.db import clear_db_assets @@ -137,7 +137,6 @@ def test_register_asset_change(self, session, mock_task_instance): asm = AssetModel(uri="test://asset1/", name="test_asset_uri", group="asset") session.add(asm) asm.scheduled_dags = [DagScheduleAssetReference(dag_id=dag.dag_id) for dag in (dag1, dag2)] - session.execute(delete(AssetDagRunQueue)) session.flush() asset_manager.register_asset_change(task_instance=mock_task_instance, asset=asset, session=session) @@ -150,7 +149,6 @@ def test_register_asset_change(self, session, mock_task_instance): ) assert session.scalar(select(func.count()).select_from(AssetDagRunQueue)) == 2 - @pytest.mark.usefixtures("clear_assets") def test_register_asset_change_with_alias( self, session, dag_maker, mock_task_instance, testing_dag_bundle ): @@ -173,7 +171,6 @@ def test_register_asset_change_with_alias( DagScheduleAssetAliasReference(alias_id=asam.id, dag_id=dag.dag_id) for dag in (consumer_dag_1, consumer_dag_2) ] - session.execute(delete(AssetDagRunQueue)) session.flush() asset = Asset(uri="test://asset1", name="test_asset_uri") @@ -199,7 +196,6 @@ def test_register_asset_change_no_downstreams(self, session, mock_task_instance) asset = Asset(uri="test://asset1", name="never_consumed") asm = AssetModel(uri="test://asset1/", name="never_consumed", group="asset") session.add(asm) - session.execute(delete(AssetDagRunQueue)) session.flush() asset_manager.register_asset_change(task_instance=mock_task_instance, asset=asset, session=session) @@ -212,23 +208,42 @@ def test_register_asset_change_no_downstreams(self, session, mock_task_instance) ) assert session.scalar(select(func.count()).select_from(AssetDagRunQueue)) == 0 - @pytest.mark.parametrize( - ("dialect_name", "expected_helper"), - [ - ("postgresql", "_queue_dagruns_nonpartitioned_conflict_update"), - ("mysql", "_queue_dagruns_nonpartitioned_mysql"), - ("sqlite", "_queue_dagruns_nonpartitioned_conflict_update"), - ], - ) - def test_queue_dagruns_routes_by_dialect(self, dialect_name, expected_helper): - """Test that _queue_dagruns routes to the dialect-appropriate queue helper.""" + def test_register_asset_change_is_atomic_on_caller_session(self, session, mock_task_instance): + """The AssetEvent is written on the caller's session: visible before commit, gone after rollback. + + Under the old side-session behaviour the event was committed independently and would be + orphaned if the caller's transaction rolled back. Registration is now atomic with the caller. + """ + asset_manager = AssetManager() + + asset = Asset(uri="test://atomic1", name="atomic_asset") + asm = AssetModel(uri="test://atomic1/", name="atomic_asset", group="asset") + session.add(asm) + session.flush() + + event = asset_manager.register_asset_change( + task_instance=mock_task_instance, asset=asset, session=session + ) + session.flush() + + # The event is written on the caller's session and carries a real id before any commit. + assert event is not None + assert event.id is not None + event_id = event.id + assert session.get(AssetEvent, event_id) is not None + + # Rolling back the caller's transaction discards the event -> no orphan row. + session.rollback() + assert session.get(AssetEvent, event_id) is None + + def test_queue_dagruns_calls_nonpartitioned_helper(self): + """`_queue_dagruns` delegates non-partitioned dags to the single non-partitioned helper.""" dag = DagModel(dag_id="dag1") session = mock.MagicMock(spec=Session) event = mock.MagicMock() with ( - mock.patch("airflow.assets.manager.get_dialect_name", return_value=dialect_name), mock.patch.object(AssetManager, "_queue_partitioned_dags"), - mock.patch.object(AssetManager, expected_helper) as mock_helper, + mock.patch.object(AssetManager, "_queue_dagruns_nonpartitioned") as mock_helper, ): AssetManager._queue_dagruns( asset_id=1, @@ -239,26 +254,45 @@ def test_queue_dagruns_routes_by_dialect(self, dialect_name, expected_helper): task_instance=None, session=session, ) - if expected_helper == "_queue_dagruns_nonpartitioned_conflict_update": - mock_helper.assert_called_once_with(1, {dag}, event, session, dialect_name) - elif expected_helper == "_queue_dagruns_nonpartitioned_mysql": - mock_helper.assert_called_once_with(1, {dag}, event, session) - else: - raise AssertionError(f"Unexpected expected_helper: {expected_helper}") + mock_helper.assert_called_once_with(1, {dag}, event, session) - def test_queue_dagruns_nonpartitioned_mysql_builds_upsert(self): - """Test that the MySQL queue path emits an INSERT ... ON DUPLICATE KEY UPDATE.""" + @pytest.mark.parametrize("dialect_name", ["postgresql", "sqlite"]) + def test_queue_dagruns_nonpartitioned_insert_or_ignore(self, dialect_name): + """On postgres/sqlite the helper emits INSERT ... ON CONFLICT DO NOTHING referencing the event.""" dag = DagModel(dag_id="dag1") session = mock.MagicMock(spec=Session) event = AssetEvent(asset_id=1) - AssetManager._queue_dagruns_nonpartitioned_mysql( - asset_id=1, dags_to_queue={dag}, event=event, session=session - ) + event.id = 99 + with mock.patch("airflow.assets.manager.get_dialect_name", return_value=dialect_name): + AssetManager._queue_dagruns_nonpartitioned( + asset_id=1, dags_to_queue={dag}, event=event, session=session + ) + + stmt, values = session.execute.call_args.args + dialect = {"postgresql": postgresql.dialect(), "sqlite": sqlite.dialect()}[dialect_name] + compiled = str(stmt.compile(dialect=dialect)).upper() + assert "ON CONFLICT" in compiled + assert "DO NOTHING" in compiled + # One ADRQ row is inserted per (dag, event), carrying the denormalized asset_id + # plus the referenced asset_event_id. + assert values == [{"asset_id": 1, "target_dag_id": "dag1", "asset_event_id": 99}] + + def test_queue_dagruns_nonpartitioned_upsert_noop_on_mysql(self): + """On MySQL the helper upserts with a no-op ON DUPLICATE KEY UPDATE (scoped to duplicate + keys, unlike INSERT IGNORE) referencing the triggering event.""" + dag = DagModel(dag_id="dag1") + session = mock.MagicMock(spec=Session) + event = AssetEvent(asset_id=1) + event.id = 99 + with mock.patch("airflow.assets.manager.get_dialect_name", return_value="mysql"): + AssetManager._queue_dagruns_nonpartitioned( + asset_id=1, dags_to_queue={dag}, event=event, session=session + ) stmt, values = session.execute.call_args.args compiled = str(stmt.compile(dialect=mysql.dialect())).upper() - assert "ON DUPLICATE KEY UPDATE" in compiled - assert values == [{"target_dag_id": "dag1"}] + assert "ON DUPLICATE KEY UPDATE ASSET_ID = ASSET_DAG_RUN_QUEUE.ASSET_ID" in compiled + assert values == [{"asset_id": 1, "target_dag_id": "dag1", "asset_event_id": 99}] def test_register_asset_change_notifies_asset_listener( self, session, mock_task_instance, testing_dag_bundle, listener_manager @@ -285,6 +319,44 @@ def test_register_asset_change_notifies_asset_listener( assert len(asset_listener.changed) == 1 assert asset_listener.changed[0].uri == asset.uri + def test_register_asset_change_defers_notifications_to_callback_sink( + self, session, mock_task_instance, testing_dag_bundle, listener_manager + ): + asset_manager = AssetManager() + asset_listener.clear() + listener_manager(asset_listener) + + bundle_name = "testing" + + asset = Asset(uri="test://asset1", name="test_asset_1") + dag1 = DagModel(dag_id="dag3", bundle_name=bundle_name) + session.add(dag1) + + asm = AssetModel(uri="test://asset1/", name="test_asset_1", group="asset") + session.add(asm) + asm.scheduled_dags = [DagScheduleAssetReference(dag_id=dag1.dag_id)] + session.flush() + + # When a callback_sink is supplied, listener notifications are collected into it + # instead of firing inline, so the caller can run them after releasing the lock. + callback_sink: list = [] + asset_manager.register_asset_change( + task_instance=mock_task_instance, + asset=asset, + session=session, + callback_sink=callback_sink, + ) + session.flush() + + assert asset_listener.changed == [] + assert callback_sink + + # Running the collected callbacks fires the listeners. + for callback in callback_sink: + callback() + assert len(asset_listener.changed) == 1 + assert asset_listener.changed[0].uri == asset.uri + def test_create_assets_notifies_asset_listener(self, session, listener_manager): asset_manager = AssetManager() asset_listener.clear() @@ -343,7 +415,7 @@ def _get_or_create_apdr(): assert len(set(ids)) == 1 assert session.scalar(select(func.count()).select_from(AssetPartitionDagRun)) == 1 - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_get_or_create_apdr_suppresses_conflicting_partition_date(self, session): """Two events resolving the same target key to different dates → suppress to None. @@ -378,7 +450,7 @@ def test_get_or_create_apdr_suppresses_conflicting_partition_date(self, session) assert second.id == first.id # same pending APDR assert second.partition_date is None # conflict suppressed, deterministic - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_get_or_create_apdr_keeps_agreeing_partition_date(self, session): """A later event carrying the same (or no) date does not trip the conflict suppression.""" asm = AssetModel(uri="test://asset1/", name="partition_asset", group="asset") @@ -406,7 +478,7 @@ def test_get_or_create_apdr_keeps_agreeing_partition_date(self, session): assert with_none.id == first.id assert with_none.partition_date == source_date - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_get_or_create_apdr_adopts_date_when_existing_is_none(self, session): """An APDR created with no date adopts a later event's carried date (not dropped).""" asm = AssetModel(uri="test://asset1/", name="partition_asset", group="asset") @@ -431,7 +503,7 @@ def test_get_or_create_apdr_adopts_date_when_existing_is_none(self, session): assert adopted.id == first.id assert adopted.partition_date == source_date - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_get_or_create_apdr_recovers_after_conflict(self, session): """Once a conflict has suppressed the date to None, a later event re-adopts a date.""" asm = AssetModel(uri="test://asset1/", name="partition_asset", group="asset") @@ -459,7 +531,7 @@ def test_get_or_create_apdr_recovers_after_conflict(self, session): assert recovered.id == first.id assert recovered.partition_date == date_2 - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_carry_partition_date_failure_degrades_to_none(self, session, dag_maker, mock_task_instance): """A mapper whose carry_partition_date raises must not abort the write. @@ -596,7 +668,6 @@ def test_register_asset_change_queues_stale_dag(self, session, mock_task_instanc # Link the Stale Dag to the Asset asm.scheduled_dags = [DagScheduleAssetReference(dag_id=stale_dag.dag_id)] - session.execute(delete(AssetDagRunQueue)) session.flush() # Register the asset change @@ -611,7 +682,7 @@ def test_register_asset_change_queues_stale_dag(self, session, mock_task_instanc queued_id = session.scalar(select(AssetDagRunQueue.target_dag_id)) assert queued_id == "stale_dag" - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partitioned_asset_event_does_not_trigger_non_partitioned_dag(self, session, mock_task_instance): """partitioned asset events (events with partition key) must not queue non-partition-aware Dags.""" asm = AssetModel(uri="test://asset/", name="test_asset", group="asset") @@ -621,7 +692,6 @@ def test_partitioned_asset_event_does_not_trigger_non_partitioned_dag(self, sess ) session.add(dag) asm.scheduled_dags = [DagScheduleAssetReference(dag_id=dag.dag_id)] - session.execute(delete(AssetDagRunQueue)) session.flush() AssetManager.register_asset_change( @@ -645,7 +715,7 @@ def test_partitioned_asset_event_does_not_trigger_non_partitioned_dag(self, sess pytest.param(6, True, id="one_over_cap_trips"), ], ) - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partition_fan_out_cap(self, session, dag_maker, mock_task_instance, cap, expect_trip): """The ``[scheduler] partition_mapper_max_downstream_keys`` cap gates fan-out. @@ -702,7 +772,7 @@ def test_partition_fan_out_cap(self, session, dag_maker, mock_task_instance, cap assert error_call.kwargs["cap_source"] == f"[scheduler] partition_mapper_max_downstream_keys={cap}" @conf_vars({("scheduler", "partition_mapper_max_downstream_keys"): "100"}) - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partition_fanout_per_mapper_override_stricter_than_global_trips( self, session, dag_maker, mock_task_instance ): @@ -749,7 +819,7 @@ def test_partition_fanout_per_mapper_override_stricter_than_global_trips( assert error_call.kwargs["cap_source"] == "max_downstream_keys=3" @conf_vars({("scheduler", "partition_mapper_max_downstream_keys"): "3"}) - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partition_fanout_per_mapper_override_looser_than_global_permits( self, session, dag_maker, mock_task_instance ): @@ -793,7 +863,7 @@ def test_partition_fanout_per_mapper_override_looser_than_global_permits( ) @conf_vars({("scheduler", "partition_mapper_max_downstream_keys"): "1"}) - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partition_fanout_per_mapper_at_cap_is_allowed(self, session, dag_maker, mock_task_instance): """Per-mapper max_downstream_keys=7 with a 7-key fanout: exactly at cap is allowed. @@ -832,7 +902,7 @@ def test_partition_fanout_per_mapper_at_cap_is_allowed(self, session, dag_maker, ) @conf_vars({("scheduler", "partition_mapper_max_downstream_keys"): "1"}) - @pytest.mark.usefixtures("clear_assets", "testing_dag_bundle") + @pytest.mark.usefixtures("testing_dag_bundle") def test_partition_fanout_per_mapper_one_over_cap_trips(self, session, dag_maker, mock_task_instance): """Per-mapper max_downstream_keys=6 with a 7-key fanout: one over cap trips the guard. diff --git a/airflow-core/tests/unit/cli/commands/test_command_deprecations.py b/airflow-core/tests/unit/cli/commands/test_command_deprecations.py index 667a2dc7ff53c..bcb2d7bf01f2b 100644 --- a/airflow-core/tests/unit/cli/commands/test_command_deprecations.py +++ b/airflow-core/tests/unit/cli/commands/test_command_deprecations.py @@ -36,6 +36,7 @@ config_command, connection_command, dag_command, + jobs_command, pool_command, provider_command, task_command, @@ -75,7 +76,7 @@ (variable_command.variables_import, "airflowctl variables import"), (asset_command.asset_materialize, "airflowctl assets materialize"), (asset_command.asset_list, "airflowctl assets list / airflowctl assets list-aliases"), - (asset_command.asset_details, "airflowctl assets get / airflowctl assets get-by-alias"), + (asset_command.asset_details, "airflowctl assets get / airflowctl assets get-alias"), (provider_command.provider_get, "airflowctl providers get"), (provider_command.providers_list, "airflowctl providers list"), (config_command.get_value, "airflowctl config get"), @@ -86,6 +87,7 @@ (task_command.task_states_for_dag_run, "airflowctl tasks states-for-dag-run"), (task_command.task_clear, "airflowctl tasks clear"), (task_command.task_failed_deps, "airflowctl tasks failed-deps"), + (jobs_command.check, "airflowctl jobs check"), ] diff --git a/airflow-core/tests/unit/cli/commands/test_connection_command.py b/airflow-core/tests/unit/cli/commands/test_connection_command.py index 2bb77b2db6874..8f285d2db0bd9 100644 --- a/airflow-core/tests/unit/cli/commands/test_connection_command.py +++ b/airflow-core/tests/unit/cli/commands/test_connection_command.py @@ -481,7 +481,7 @@ def setup_method(self): "new0-json", f"--conn-json={TEST_JSON}", ], - "Successfully added `conn_id`=new0-json : postgres://airflow:******@host:5432/airflow", + "Successfully added `conn_id`=new0-json", { "conn_type": "postgres", "description": "new0-json description", @@ -503,7 +503,7 @@ def setup_method(self): f"--conn-uri={TEST_URL}", "--conn-description=new0 description", ], - "Successfully added `conn_id`=new0 : postgresql://airflow:airflow@host:5432/airflow", + "Successfully added `conn_id`=new0", { "conn_type": "postgres", "description": "new0 description", @@ -525,7 +525,7 @@ def setup_method(self): f"--conn-uri={TEST_URL}", "--conn-description=new1 description", ], - "Successfully added `conn_id`=new1 : postgresql://airflow:airflow@host:5432/airflow", + "Successfully added `conn_id`=new1", { "conn_type": "postgres", "description": "new1 description", @@ -548,7 +548,7 @@ def setup_method(self): "--conn-extra", '{"extra": "yes"}', ], - "Successfully added `conn_id`=new2 : postgresql://airflow:airflow@host:5432/airflow", + "Successfully added `conn_id`=new2", { "conn_type": "postgres", "description": None, @@ -573,7 +573,7 @@ def setup_method(self): "--conn-description", "new3 description", ], - "Successfully added `conn_id`=new3 : postgresql://airflow:airflow@host:5432/airflow", + "Successfully added `conn_id`=new3", { "conn_type": "postgres", "description": "new3 description", @@ -600,7 +600,7 @@ def setup_method(self): "--conn-schema=airflow", "--conn-description= new4 description ", ], - "Successfully added `conn_id`=new4 : hive_metastore://airflow:******@host:9083/airflow", + "Successfully added `conn_id`=new4", { "conn_type": "hive_metastore", "description": " new4 description ", @@ -626,7 +626,7 @@ def setup_method(self): '{"extra": "yes"}', "--conn-description=new5 description", ], - "Successfully added `conn_id`=new5 : google_cloud_platform://:@:", + "Successfully added `conn_id`=new5", { "conn_type": "google_cloud_platform", "description": "new5 description", @@ -642,7 +642,7 @@ def setup_method(self): ), pytest.param( ["connections", "add", "new6", "--conn-uri", "aws://?region_name=foo-bar-1"], - "Successfully added `conn_id`=new6 : aws://?region_name=foo-bar-1", + "Successfully added `conn_id`=new6", { "conn_type": "aws", "description": None, @@ -658,7 +658,7 @@ def setup_method(self): ), pytest.param( ["connections", "add", "new7", "--conn-uri", "aws://@/?region_name=foo-bar-1"], - "Successfully added `conn_id`=new7 : aws://@/?region_name=foo-bar-1", + "Successfully added `conn_id`=new7", { "conn_type": "aws", "description": None, diff --git a/airflow-core/tests/unit/core/test_configuration.py b/airflow-core/tests/unit/core/test_configuration.py index 87a083c7e02ca..815574a30e137 100644 --- a/airflow-core/tests/unit/core/test_configuration.py +++ b/airflow-core/tests/unit/core/test_configuration.py @@ -1982,6 +1982,25 @@ def test_provider_sections_do_not_overlap_with_core(): ) +@pytest.mark.parametrize( + ("sqlite_version_info", "expect_error"), + [ + ((3, 15, 0), False), + ((3, 14, 9), True), + ], +) +def test_validate_sqlite3_version(sqlite_version_info, expect_error, monkeypatch): + """_validate_sqlite3_version compares against sqlite3.sqlite_version_info, not a parsed string.""" + monkeypatch.setenv("AIRFLOW__DATABASE__SQL_ALCHEMY_CONN", "sqlite:////tmp/airflow.db") + test_conf = AirflowConfigParser() + with mock.patch("sqlite3.sqlite_version_info", sqlite_version_info): + if expect_error: + with pytest.raises(AirflowConfigException, match="SQLite C library too old"): + test_conf._validate_sqlite3_version() + else: + test_conf._validate_sqlite3_version() + + @skip_if_force_lowest_dependencies_marker class TestProviderConfigPriority: """Tests that conf.get and conf.has_option respect provider metadata and cfg fallbacks with correct priority.""" diff --git a/airflow-core/tests/unit/dag_processing/test_dagbag.py b/airflow-core/tests/unit/dag_processing/test_dagbag.py index 0ea0a29fc145b..285a00bc25cb9 100644 --- a/airflow-core/tests/unit/dag_processing/test_dagbag.py +++ b/airflow-core/tests/unit/dag_processing/test_dagbag.py @@ -185,7 +185,7 @@ def test_executor_validation_failure_with_team(self, mock_lookup, mock_manager_c "Task 'task1' specifies executor 'invalid.executor', which is not available " "for team 'test_team' (the team associated with DAG 'test-dag') or as a global executor. " "Make sure 'invalid.executor' is configured for team 'test_team' or globally in your " - "[core] executors configuration, or update the task's executor to use one of the " + "[core] executor configuration, or update the task's executor to use one of the " "configured executors for team 'test_team' or available global executors." ), ): @@ -204,7 +204,7 @@ def test_executor_validation_failure_no_team(self, mock_lookup): UnknownExecutorException, match=re.escape( "Task 'task1' specifies executor 'invalid.executor', which is not available. " - "Make sure it is listed in your [core] executors configuration, or update the task's " + "Make sure it is listed in your [core] executor configuration, or update the task's " "executor to use one of the configured executors." ), ): @@ -258,7 +258,7 @@ def test_global_executor_fallback_failure(self, mock_lookup, mock_manager_class) "Task 'task1' specifies executor 'unknown.executor', which is not available " "for team 'test_team' (the team associated with DAG 'test-dag') or as a global executor. " "Make sure 'unknown.executor' is configured for team 'test_team' or globally in your " - "[core] executors configuration, or update the task's executor to use one of the " + "[core] executor configuration, or update the task's executor to use one of the " "configured executors for team 'test_team' or available global executors." ), ): @@ -330,7 +330,7 @@ def test_validate_executor_field_executor_not_configured(): UnknownExecutorException, match=re.escape( "Task 't1' specifies executor 'test.custom.executor', which is not available. " - "Make sure it is listed in your [core] executors configuration, or update the task's " + "Make sure it is listed in your [core] executor configuration, or update the task's " "executor to use one of the configured executors." ), ): diff --git a/airflow-core/tests/unit/dag_processing/test_manager.py b/airflow-core/tests/unit/dag_processing/test_manager.py index f1fb4b66c81fd..4599c8ea58342 100644 --- a/airflow-core/tests/unit/dag_processing/test_manager.py +++ b/airflow-core/tests/unit/dag_processing/test_manager.py @@ -576,7 +576,6 @@ def test_files_sorted_random_seeded_by_host(self): assert manager._file_queue == expected # Verify running it again produces same order - manager._files = [] manager.prepare_file_queue(known_files=known_files) assert manager._file_queue == expected @@ -1016,7 +1015,6 @@ def test_scan_stale_dags(self, session): run_count=1, last_num_of_db_queries=1, ) - manager._files = [test_dag_path] manager._file_stats[test_dag_path] = stat active_dag_count = session.scalar( @@ -1046,6 +1044,73 @@ def test_scan_stale_dags(self, session): # SerializedDagModel gives history about Dags assert serialized_dag_count == 1 + @pytest.mark.usefixtures("testing_dag_bundle") + def test_scan_stale_dags_deactivates_zip_packaged_dags(self, session, test_zip_path): + """ + Ensure that zip-packaged DAGs are marked inactive when the file is parsed but the + DagModel.last_parsed_time is not updated, testing fallback to the parent path when + comparing DAG.relative_fileloc to last_parsed entries. + """ + manager = DagFileProcessorManager( + max_runs=1, + processor_timeout=10 * 60, + ) + bundle = MagicMock() + bundle.name = "testing" + manager._dag_bundles = [bundle] + + test_dag_path = DagFileInfo( + rel_path=Path("test_zip.zip"), + bundle_path=Path(test_zip_path).parent, + bundle_name="testing", + ) + dagbag = DagBag( + test_dag_path.absolute_path, + bundle_path=test_dag_path.bundle_path, + ) + + # Add stale DAG to the DB + dag = dagbag.get_dag("test_zip_dag") + sync_dag_to_db(dag, session=session) + + # Add DAG to the file_parsing_stats + stat = DagFileStat( + num_dags=1, + import_errors=0, + last_finish_time=timezone.utcnow() + timedelta(hours=1), + last_duration=1, + run_count=1, + last_num_of_db_queries=1, + ) + manager._file_stats[test_dag_path] = stat + + active_dag_count = session.scalar( + select(func.count(DagModel.dag_id)).where( + DagModel.dag_id == "test_zip_dag", + ~DagModel.is_stale, + DagModel.relative_fileloc == str(test_dag_path.rel_path / "test_zip.py"), + ) + ) + assert active_dag_count == 1 + + manager._scan_stale_dags() + + active_dag_count = session.scalar( + select(func.count(DagModel.dag_id)).where( + DagModel.dag_id == "test_zip_dag", + ~DagModel.is_stale, + DagModel.relative_fileloc == str(test_dag_path.rel_path / "test_zip.py"), + ) + ) + assert active_dag_count == 0 + + serialized_dag_count = session.scalar( + select(func.count(SerializedDagModel.dag_id)).where(SerializedDagModel.dag_id == dag.dag_id) + ) + # Deactivating the DagModel should not delete the SerializedDagModel + # SerializedDagModel gives history about Dags + assert serialized_dag_count == 1 + @pytest.mark.usefixtures("testing_dag_bundle") def test_deactivate_stale_dags_marks_dags_in_inactive_bundles(self, session): """Dags whose bundle is no longer active should be marked stale even without a parse signal.""" @@ -3186,6 +3251,39 @@ def test_refresh_dag_bundles_get_bundle_state_failure_skips_bundle(self): bundle.refresh.assert_not_called() + def test_refresh_dag_bundles_initialize_non_airflow_exception_skips_bundle(self): + """ + A bundle whose initialize() raises a non AirflowException must be skipped, not + left to propagate and abort refresh for every other bundle. + """ + manager = DagFileProcessorManager(max_runs=1) + failing_bundle = self._make_refresh_bundle() + failing_bundle.name = "failing_bundle" + failing_bundle.is_initialized = False + failing_bundle.initialize.side_effect = FileNotFoundError("Repository path not found") + + healthy_bundle = self._make_refresh_bundle() + healthy_bundle.name = "healthy_bundle" + + manager._dag_bundles = [failing_bundle, healthy_bundle] + manager._force_refresh_bundles = set() + + with ( + mock.patch.object( + manager, "get_bundle_state", return_value=BundleState(last_refreshed=None, version=None) + ), + mock.patch.object(manager, "update_bundle_state"), + mock.patch.object(manager, "_find_files_in_bundle", return_value=[]), + mock.patch.object(manager, "deactivate_deleted_dags"), + mock.patch.object(manager, "clear_orphaned_import_errors"), + mock.patch.object(manager, "handle_removed_files"), + mock.patch.object(manager, "_resort_file_queue"), + mock.patch.object(manager, "_add_new_files_to_queue"), + ): + manager._refresh_dag_bundles({}) + + healthy_bundle.refresh.assert_called_once() + def test_refresh_dag_bundles_update_bundle_state_failure_still_scans_files(self): """A failure in update_bundle_state() logs but does not skip file scanning. diff --git a/airflow-core/tests/unit/jobs/test_scheduler_job.py b/airflow-core/tests/unit/jobs/test_scheduler_job.py index 53ea5bc933f3f..440e6e2648c46 100644 --- a/airflow-core/tests/unit/jobs/test_scheduler_job.py +++ b/airflow-core/tests/unit/jobs/test_scheduler_job.py @@ -47,7 +47,7 @@ from airflow._shared.observability.metrics.base_stats_logger import StatsLogger from airflow._shared.timezones import timezone from airflow.api_fastapi.auth.tokens import JWTGenerator -from airflow.assets.manager import AssetManager, _create_asset_event +from airflow.assets.manager import AssetManager from airflow.callbacks.callback_requests import ( DagCallbackRequest, DagRunContext, @@ -342,6 +342,14 @@ def set_instance_attrs(self) -> Generator: yield self.null_exec = None + @pytest.fixture + def team_bundle(self, testing_team, testing_dag_bundle, session): + team = session.merge(testing_team) + bundle = session.scalar(select(DagBundleModel).where(DagBundleModel.name == "testing")) + bundle.teams.append(team) + session.flush() + return bundle + @pytest.fixture def mock_executors(self): mock_jwt_generator = MagicMock(spec=JWTGenerator) @@ -5707,16 +5715,14 @@ def test_create_dag_runs_assets(self, session, dag_maker): timestamp=base + timedelta(seconds=2), ) session.add_all([event1, event2]) + session.flush() # assign event ids so the ADRQ rows can reference them session = dag_maker.session session.add_all( [ - AssetDagRunQueue( - asset_id=asset1_id, target_dag_id=dag2.dag_id, created_at=base + timedelta(hours=1) - ), - AssetDagRunQueue( - asset_id=asset1_id, target_dag_id=dag3.dag_id, created_at=base + timedelta(hours=1) - ), + AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag2.dag_id, asset_event_id=event1.id), + AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag3.dag_id, asset_event_id=event1.id), + AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag3.dag_id, asset_event_id=event2.id), ] ) session.flush() @@ -5773,28 +5779,21 @@ def dict_from_obj(obj): def test_new_asset_triggered_dag_backlog_gated_by_catchup( self, catchup, expects_old_event, session, dag_maker ): - """Reproduces #39456: catchup gates whether a new asset-triggered Dag replays the - pre-creation backlog. With catchup off (the default) it only consumes events after it - started scheduling on the asset; with catchup on it replays the full history.""" + """catchup gates whether a newly-subscribed asset-triggered Dag consumes its backlog. + + With catchup off (the default) the Dag consumes only events with a queue row (i.e. those + emitted after it began scheduling on the asset). With catchup on, the first triggered run + also consumes the pre-subscription backlog -- every not-yet-consumed event for the Dag's + assets -- selected directly by the scheduler (no queue row required, no time window). + """ asset = Asset(uri="test://asset-historical", name="hist_asset", group="test_group") - # Producer Dag + run that the asset events are sourced from. + # Producer + a historical event that exists BEFORE any consumer subscribes. with dag_maker(dag_id="historical-producer", start_date=timezone.utcnow(), session=session): BashOperator(task_id="task", bash_command="echo 1", outlets=[asset]) producer_run = dag_maker.create_dagrun(run_id="producer-run") - asset_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset.uri)) - # Consumer Dag created now; its schedule reference's created_at is the cut-off. - with dag_maker(dag_id="historical-consumer", schedule=[asset], catchup=catchup): - pass - consumer_dag = dag_maker.dag - reference_created_at = session.scalar( - select(DagScheduleAssetReference.created_at).where( - DagScheduleAssetReference.dag_id == consumer_dag.dag_id - ) - ) - def _make_event(timestamp): return AssetEvent( asset_id=asset_id, @@ -5805,18 +5804,28 @@ def _make_event(timestamp): timestamp=timestamp, ) - old_event = _make_event(reference_created_at - timedelta(days=1)) - new_event = _make_event(reference_created_at + timedelta(seconds=1)) - session.add_all([old_event, new_event]) - # Trigger time after both events so neither is excluded by the upper bound. + old_event = _make_event(timezone.utcnow() - timedelta(days=1)) + session.add(old_event) + session.commit() + + # Consumer subscribes now: catchup=True backfills a queue row for old_event (the + # pre-subscription backlog); catchup=False backfills nothing. + with dag_maker(dag_id="historical-consumer", schedule=[asset], catchup=catchup, session=session): + pass + consumer_dag = dag_maker.dag + + # A post-subscription event always gets its own queue row. + new_event = _make_event(timezone.utcnow()) + session.add(new_event) + session.flush() session.add( AssetDagRunQueue( asset_id=asset_id, target_dag_id=consumer_dag.dag_id, - created_at=reference_created_at + timedelta(hours=1), + asset_event_id=new_event.id, ) ) - session.flush() + session.commit() scheduler_job = Job() self.job_runner = SchedulerJobRunner(job=scheduler_job, executors=[self.null_exec]) @@ -5828,6 +5837,78 @@ def _make_event(timestamp): expected = {new_event.id} | ({old_event.id} if expects_old_event else set()) assert {e.id for e in created_run.consumed_asset_events} == expected + @pytest.mark.need_serialized_dag + def test_asset_events_out_of_order_are_both_consumed(self, session, dag_maker): + """Regression test for GH-54659. + + Two events for the same asset can become visible out of timestamp order (for example a + long-running producer commits an "older" event after a "newer" one). Under the old + created_at/timestamp watermark the older event could be stranded below the watermark and + never consumed. With consume-by-reference every event referenced by an ADRQ row is + consumed, regardless of the order in which the timestamps became visible. + """ + asset = Asset(uri="test://asset-ooo", name="ooo_asset", group="test_group") + + with dag_maker(dag_id="ooo-producer", start_date=timezone.utcnow(), session=session): + BashOperator(task_id="task", bash_command="echo 1", outlets=[asset]) + producer_run = dag_maker.create_dagrun(run_id="producer-run") + + asset_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset.uri)) + + with dag_maker(dag_id="ooo-consumer", schedule=[asset]): + pass + consumer_dag = dag_maker.dag + + base = timezone.utcnow() + + def _make_event(timestamp): + return AssetEvent( + asset_id=asset_id, + source_task_id="task", + source_dag_id=producer_run.dag_id, + source_run_id=producer_run.run_id, + source_map_index=-1, + timestamp=timestamp, + ) + + # The "newer" event is registered (and gets its lower id) BEFORE the "older" one, so + # insertion order and timestamp order disagree. + newer_event = _make_event(base + timedelta(seconds=10)) + session.add(newer_event) + session.flush() + older_event = _make_event(base + timedelta(seconds=1)) + session.add(older_event) + session.flush() + + session.add_all( + [ + AssetDagRunQueue( + asset_id=asset_id, target_dag_id=consumer_dag.dag_id, asset_event_id=newer_event.id + ), + AssetDagRunQueue( + asset_id=asset_id, target_dag_id=consumer_dag.dag_id, asset_event_id=older_event.id + ), + ] + ) + session.flush() + + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(job=scheduler_job, executors=[self.null_exec]) + with create_session() as session: + self.job_runner._create_dagruns_for_dags(session, session) + + created_run = session.scalars(select(DagRun).where(DagRun.dag_id == consumer_dag.dag_id)).one() + assert created_run.state == State.QUEUED + # Neither event is stranded: both are consumed by the single run. + assert {e.id for e in created_run.consumed_asset_events} == {newer_event.id, older_event.id} + # All ADRQ rows for the dag are cleared once consumed. + assert ( + session.scalars( + select(AssetDagRunQueue).where(AssetDagRunQueue.target_dag_id == consumer_dag.dag_id) + ).all() + == [] + ) + @pytest.mark.need_serialized_dag def test_create_dag_runs_asset_triggered_skips_stale_triggered_date(self, session, dag_maker): asset = Asset(uri="test://asset-for-stale-trigger-date", name="asset-for-stale-trigger-date") @@ -5836,8 +5917,12 @@ def test_create_dag_runs_asset_triggered_skips_stale_triggered_date(self, sessio dag_model = dag_maker.dag_model asset_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset.uri)) - queued_at = timezone.utcnow() - session.add(AssetDagRunQueue(target_dag_id=dag_model.dag_id, asset_id=asset_id, created_at=queued_at)) + event = AssetEvent(asset_id=asset_id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + session.add( + AssetDagRunQueue(target_dag_id=dag_model.dag_id, asset_id=asset_id, asset_event_id=event.id) + ) session.flush() # Simulate another scheduler consuming ADRQ rows after we computed triggered_date_by_dag. @@ -5865,21 +5950,20 @@ def test_create_dag_runs_asset_triggered_deletes_only_selected_adrq_rows( dag_model = dag_maker.dag_model asset_1_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset_1.name)) asset_2_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset_2.name)) + event_1 = AssetEvent(asset_id=asset_1_id, timestamp=timezone.utcnow()) + event_2 = AssetEvent(asset_id=asset_2_id, timestamp=timezone.utcnow()) + session.add_all([event_1, event_2]) + session.flush() session.add_all( [ - AssetEvent( - asset_id=asset_1_id, - timestamp=timezone.utcnow(), - ), - # The ADRQ that should triggers the Dag run creation + # The ADRQ that should trigger the Dag run creation AssetDagRunQueue( - asset_id=asset_1_id, target_dag_id=dag_model.dag_id, created_at=timezone.utcnow() + asset_id=asset_1_id, target_dag_id=dag_model.dag_id, asset_event_id=event_1.id ), - AssetEvent(asset_id=asset_2_id, timestamp=timezone.utcnow()), - # The ADRQ that arrives after the Dag run creation but before ADRQ clean up - # This situation is simulated by _lock_only_selected_asset below + # The ADRQ that arrives after the Dag run creation but before ADRQ clean up. + # This situation is simulated by _lock_only_selected_asset below. AssetDagRunQueue( - asset_id=asset_2_id, target_dag_id=dag_model.dag_id, created_at=timezone.utcnow() + asset_id=asset_2_id, target_dag_id=dag_model.dag_id, asset_event_id=event_2.id ), ] ) @@ -5946,23 +6030,23 @@ def create_asset_events(sleep): dag = session.get(DagModel, consumer_dag_id) now = timezone.utcnow() asset_manager = AssetManager() - asset_event = _create_asset_event(session=session, asset_id=asset_id, timestamp=now) + # The event is now created inline on the caller's session (atomic), rather than in + # a side session, so build it directly here. + asset_event = AssetEvent(asset_id=asset_id, timestamp=now) + session.add(asset_event) + session.flush() + event_id = asset_event.id time.sleep(sleep) # widen the race window between event creation and queueing - dialect_name = inspect(session.get_bind()).dialect.name - if dialect_name in ("postgresql", "sqlite"): - asset_manager._queue_dagruns_nonpartitioned_conflict_update( - asset_id=asset_id, - dags_to_queue=[dag], - event=asset_event, - session=session, - dialect_name=dialect_name, - ) - elif dialect_name == "mysql": - asset_manager._queue_dagruns_nonpartitioned_mysql( - asset_id=asset_id, dags_to_queue=[dag], event=asset_event, session=session - ) + # A single dialect-agnostic helper now performs insert-or-ignore keyed on + # (target_dag_id, asset_event_id). + asset_manager._queue_dagruns_nonpartitioned( + asset_id=asset_id, + dags_to_queue={dag}, + event=asset_event, + session=session, + ) - return asset_event.id, now.isoformat() + return event_id, now.isoformat() with ( ThreadPoolExecutor(max_workers=3) as executor, @@ -6045,7 +6129,9 @@ def test_create_dag_runs_asset_alias_with_asset_event_attached(self, session, da session = dag_maker.session session.add_all( [ - AssetDagRunQueue(asset_id=asset1_id, target_dag_id=consumer_dag.dag_id), + AssetDagRunQueue( + asset_id=asset1_id, target_dag_id=consumer_dag.dag_id, asset_event_id=event.id + ), ] ) session.flush() @@ -6119,9 +6205,11 @@ def test_asset_triggered_dagruns_respects_team_name( source_map_index=-1, ) session.add(event) - # flush here to ensure event timestamp is before the ADRQ created_at timestamp + # flush here to assign the event id referenced by the ADRQ row session.flush() - session.add(AssetDagRunQueue(asset_id=asset_id, target_dag_id=f"consumer_{suffix}")) + session.add( + AssetDagRunQueue(asset_id=asset_id, target_dag_id=f"consumer_{suffix}", asset_event_id=event.id) + ) session.flush() with conf_vars({("core", "multi_team"): multi_team}): @@ -6187,26 +6275,42 @@ def test_no_create_dag_runs_when_dag_disabled(self, session, dag_maker, disable, ) session.flush() assert [e.source_run_id for e in session.scalars(ase_q)] == [dr1.run_id, dr2.run_id] - assert len(session.scalars(adrq_q).all()) == 1 - assert session.scalars(adrq_q).one().target_dag_id == "consumer" + # ADRQ rows are per asset event now. A stale dag still enqueues while disabled (asserted + # above), so both events stay queued; a paused dag never enqueued the first event, so only + # the second remains. + expected_adrqs = 2 if "is_stale" in disable else 1 + adrqs = session.scalars(adrq_q).all() + assert len(adrqs) == expected_adrqs + assert all(adrq.target_dag_id == "consumer" for adrq in adrqs) @pytest.mark.need_serialized_dag - def test_no_create_dag_runs_when_no_asset_event(self, session: Session, dag_maker, caplog): + def test_no_create_dag_runs_when_asset_event_already_consumed(self, session: Session, dag_maker, caplog): asset = Asset(name="test_asset") with dag_maker(dag_id="consumer", schedule=asset, session=session): pass dag_model = dag_maker.dag_model asset_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset.uri)) - # Simulate an ADRQ row whose matching asset events were already consumed by an earlier DagRun. - # The ADRQ should be cleaned up even when no new DagRun is created, to prevent stale ADRQ - # rows from accumulating and causing infinite scheduler loops. - adrq = AssetDagRunQueue( - asset_id=asset_id, target_dag_id=dag_model.dag_id, created_at=timezone.utcnow() + + # An event that has already been consumed by an earlier DagRun of this dag. + event = AssetEvent(asset_id=asset_id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + prior_run = dag_maker.create_dagrun( + run_id="prior-consuming-run", + logical_date=DEFAULT_DATE, + data_interval=(DEFAULT_DATE, DEFAULT_DATE), ) - session.add(adrq) + prior_run.consumed_asset_events.append(event) session.flush() - adrq.created_at = timezone.utcnow() + timedelta(seconds=1) - session.merge(adrq) + + # A stale ADRQ that still references the already-consumed event. It should be cleaned up + # even when no new DagRun is created, to prevent stale ADRQ rows from accumulating and + # causing infinite scheduler loops. + session.add( + AssetDagRunQueue(asset_id=asset_id, target_dag_id=dag_model.dag_id, asset_event_id=event.id) + ) + session.flush() + with caplog.at_level("INFO"): scheduler_job = Job() self.job_runner = SchedulerJobRunner(job=scheduler_job, executors=[MockExecutor(do_update=False)]) @@ -6214,9 +6318,11 @@ def test_no_create_dag_runs_when_no_asset_event(self, session: Session, dag_make dag_models=[dag_model], session=session, ) - dr = session.scalars(select(DagRun).where(DagRun.dag_id == dag_model.dag_id)).one_or_none() + assert "No DagRun created" in caplog.text - assert dr is None + # No *new* DagRun is created; only the pre-existing consuming run remains. + runs = session.scalars(select(DagRun).where(DagRun.dag_id == dag_model.dag_id)).all() + assert [r.run_id for r in runs] == ["prior-consuming-run"] _adrq = session.scalars( select(AssetDagRunQueue).where( AssetDagRunQueue.asset_id == asset_id, AssetDagRunQueue.target_dag_id == dag_model.dag_id @@ -8115,6 +8221,58 @@ def test_awaiting_input_timeout_without_defaults_fails(self, dag_maker): assert ti.next_method == "execute_complete" assert ti.next_kwargs["event"]["error_type"] == "timeout" + def test_awaiting_input_timeout_sweep_survives_unusable_next_kwargs(self, dag_maker): + """The sweep finishes its batch even when one task's stored kwargs cannot be read.""" + session = settings.Session() + with dag_maker( + dag_id="test_awaiting_input_bad_kwargs", + start_date=DEFAULT_DATE, + schedule="@once", + session=session, + ): + EmptyOperator(task_id="dummy1") + dr_bad = dag_maker.create_dagrun() + dr_good = dag_maker.create_dagrun( + run_id="good", logical_date=DEFAULT_DATE + datetime.timedelta(seconds=1) + ) + ti_bad = dr_bad.get_task_instance("dummy1", session=session) + ti_good = dr_good.get_task_instance("dummy1", session=session) + for ti in (ti_bad, ti_good): + ti.state = State.AWAITING_INPUT + ti.trigger_timeout = timezone.utcnow() - datetime.timedelta(seconds=60) + ti.next_method = "execute_complete" + ti.next_kwargs = {} + # Stored kwargs that no longer decode into a dict (here: a class name that is not allowed + # for deserialization, which the legacy fallback cannot read either). + ti_bad.next_kwargs = {"__classname__": "not.allowed.Thing", "__version__": 1, "__data__": {}} + session.add( + HITLDetail( + ti_id=ti_good.id, + options=["Approve", "Reject"], + subject="approve?", + defaults=["Approve"], + multiple=False, + params={}, + ) + ) + session.flush() + + self.job_runner = SchedulerJobRunner(job=Job()) + mock_log = mock.MagicMock(spec=logging.Logger) + with mock.patch.object(SchedulerJobRunner, "log", mock_log): + self.job_runner.check_awaiting_input_timeouts(session=session) + + session.refresh(ti_bad) + session.refresh(ti_good) + assert ti_bad.state == State.SCHEDULED + assert ti_bad.next_method == "__fail__" + assert "error" in ti_bad.next_kwargs + assert ti_good.state == State.SCHEDULED + assert ti_good.next_method == "execute_complete" + assert ti_good.next_kwargs["event"]["chosen_options"] == ["Approve"] + # The one it could not resume is reported as such rather than counted as resolved. + assert mock_log.info.call_args.args[1:] == (1, 0, 1) + def test_retry_on_db_error_when_update_timeout_triggers(self, dag_maker, testing_dag_bundle, session): """ Tests that it will retry on DB error like deadlock when updating timeout triggers. @@ -9635,6 +9793,147 @@ def test_dag_timeout_notifies_with_timed_out_msg(self, mock_get_listener_manager assert call_args.kwargs["msg"] == "timed_out" assert call_args.kwargs["dag_run"] == dag_run + @conf_vars({("core", "multi_team"): "true"}) + @mock.patch("airflow.models.dagrun.get_listener_manager") + def test_dag_start_notifies_listener_with_team_name( + self, mock_get_listener_manager, dag_maker, session, team_bundle + ): + """Test that on_dag_run_running receives dag_run with _team_name set.""" + mock_listener_manager = MagicMock() + mock_get_listener_manager.return_value = mock_listener_manager + + with dag_maker(dag_id="test_dag_start_team", bundle_name="testing", session=session): + EmptyOperator(task_id="test_task") + + dag_maker.create_dagrun(run_id="test_run", state=DagRunState.QUEUED) + session.commit() + + mock_executor = MagicMock() + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(scheduler_job, executors=[mock_executor]) + + self.job_runner._start_queued_dagruns(session) + + mock_listener_manager.hook.on_dag_run_running.assert_called_once() + call_args = mock_listener_manager.hook.on_dag_run_running.call_args + assert call_args.kwargs["dag_run"]._team_name == "testing" + + @conf_vars({("core", "multi_team"): "true"}) + @time_machine.travel(DEFAULT_DATE, tick=False) + @mock.patch("airflow.models.dagrun.get_listener_manager") + def test_dag_timeout_notifies_listener_with_team_name( + self, mock_get_listener_manager, dag_maker, session, team_bundle + ): + """Test that on_dag_run_failed receives dag_run with _team_name set when a DAG times out.""" + mock_listener_manager = MagicMock() + mock_get_listener_manager.return_value = mock_listener_manager + + with dag_maker( + dag_id="test_dag_timeout_team", + bundle_name="testing", + session=session, + dagrun_timeout=timedelta(seconds=60), + ): + EmptyOperator(task_id="test_task") + + dag_run = dag_maker.create_dagrun(run_id="test_run", state=DagRunState.RUNNING) + # We set it to double dagrun timeout so the timeout path is taken. + dag_run.start_date = DEFAULT_DATE - timedelta(seconds=120) + session.merge(dag_run) + session.commit() + + mock_executor = MagicMock() + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(scheduler_job, executors=[mock_executor]) + + self.job_runner._schedule_dag_run(dag_run, session) + + mock_listener_manager.hook.on_dag_run_failed.assert_called_once() + call_args = mock_listener_manager.hook.on_dag_run_failed.call_args + assert call_args.kwargs["msg"] == "timed_out" + assert call_args.kwargs["dag_run"]._team_name == "testing" + + @conf_vars({("core", "multi_team"): "true"}) + @mock.patch("airflow.models.dagrun.get_listener_manager") + def test_dag_success_notifies_listener_with_team_name( + self, mock_get_listener_manager, dag_maker, session, team_bundle + ): + """Test that on_dag_run_success receives dag_run with _team_name set.""" + mock_listener_manager = MagicMock() + mock_get_listener_manager.return_value = mock_listener_manager + + with dag_maker(dag_id="test_dag_success_team", bundle_name="testing", session=session): + EmptyOperator(task_id="test_task") + + dag_run = dag_maker.create_dagrun() + ti = dag_run.get_task_instance("test_task") + ti.set_state(TaskInstanceState.SUCCESS, session=session) + + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(scheduler_job, executors=[MockExecutor(do_update=False)]) + + self.job_runner._do_scheduling(session) + + mock_listener_manager.hook.on_dag_run_success.assert_called_once() + call_args = mock_listener_manager.hook.on_dag_run_success.call_args + assert call_args.kwargs["dag_run"]._team_name == "testing" + + @conf_vars({("core", "multi_team"): "true"}) + @mock.patch("airflow.models.dagrun.get_listener_manager") + def test_dag_failure_notifies_listener_with_team_name( + self, mock_get_listener_manager, dag_maker, session, team_bundle + ): + """Test that on_dag_run_failed receives dag_run with _team_name set.""" + mock_listener_manager = MagicMock() + mock_get_listener_manager.return_value = mock_listener_manager + + with dag_maker(dag_id="test_dag_failure_team", bundle_name="testing", session=session): + EmptyOperator(task_id="test_task") + + dag_run = dag_maker.create_dagrun() + ti = dag_run.get_task_instance("test_task") + ti.set_state(TaskInstanceState.FAILED, session=session) + + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(scheduler_job, executors=[MockExecutor(do_update=False)]) + + self.job_runner._do_scheduling(session) + + mock_listener_manager.hook.on_dag_run_failed.assert_called_once() + call_args = mock_listener_manager.hook.on_dag_run_failed.call_args + assert call_args.kwargs["dag_run"]._team_name == "testing" + + @conf_vars({("core", "multi_team"): "true"}) + @time_machine.travel(DEFAULT_DATE, tick=False) + @mock.patch("airflow.models.dagrun.get_listener_manager") + def test_dag_paused_success_notifies_listener_with_team_name( + self, mock_get_listener_manager, dag_maker, session, team_bundle + ): + """Test that on_dag_run_success receives dag_run with _team_name set for paused DAGs.""" + mock_listener_manager = MagicMock() + mock_get_listener_manager.return_value = mock_listener_manager + + with dag_maker(dag_id="test_dag_paused_team", bundle_name="testing", session=session) as dag: + EmptyOperator(task_id="test_task") + + dag_run = dag_maker.create_dagrun() + dag_run.last_scheduling_decision = DEFAULT_DATE - timedelta(minutes=1) + ti = dag_run.get_task_instance("test_task") + ti.set_state(TaskInstanceState.SUCCESS, session=session) + dm = DagModel.get_dagmodel(dag.dag_id, session=session) + dm.is_paused = True + session.flush() + + mock_executor = MagicMock() + scheduler_job = Job() + self.job_runner = SchedulerJobRunner(scheduler_job, executors=[mock_executor]) + + self.job_runner._update_dag_run_state_for_paused_dags(session=session) + + mock_listener_manager.hook.on_dag_run_success.assert_called_once() + call_args = mock_listener_manager.hook.on_dag_run_success.call_args + assert call_args.kwargs["dag_run"]._team_name == "testing" + @mock.patch("airflow.models.Deadline.handle_miss") def test_process_expired_deadlines(self, mock_handle_miss, session, dag_maker): """Verify all expired and unhandled deadlines (and only those) are processed by the scheduler.""" diff --git a/airflow-core/tests/unit/migrations/test_0129_add_indexes_on_serialized_dag_and_dag_code.py b/airflow-core/tests/unit/migrations/test_0129_add_indexes_on_serialized_dag_and_dag_code.py new file mode 100644 index 0000000000000..4774a7711dd82 --- /dev/null +++ b/airflow-core/tests/unit/migrations/test_0129_add_indexes_on_serialized_dag_and_dag_code.py @@ -0,0 +1,55 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import pytest +import sqlalchemy as sa + +from airflow import settings +from airflow.utils.db import downgrade, upgradedb + +pytestmark = pytest.mark.db_test + +_REVISION = "3c525f44bea8" +_DOWN_REVISION = "b2f1a9c7d4e0" + +_EXPECTED_INDEXES = { + "idx_serialized_dag_dag_id_created_at": ("serialized_dag", ["dag_id", "created_at"]), + "idx_dag_code_dag_id_last_updated": ("dag_code", ["dag_id", "last_updated"]), +} + + +class TestMigration0124AddDagIdIndexes: + @pytest.fixture(autouse=True) + def _restore_head(self): + yield + upgradedb() + + @staticmethod + def _get_indexes(table): + with settings.engine.connect() as conn: + return {ix["name"]: ix["column_names"] for ix in sa.inspect(conn).get_indexes(table)} + + def test_upgrade_creates_indexes_and_downgrade_drops_them(self): + downgrade(to_revision=_DOWN_REVISION) + for index, (table, _) in _EXPECTED_INDEXES.items(): + assert index not in self._get_indexes(table) + + upgradedb(to_revision=_REVISION) + for index, (table, columns) in _EXPECTED_INDEXES.items(): + assert self._get_indexes(table).get(index) == columns diff --git a/airflow-core/tests/unit/models/test_dag.py b/airflow-core/tests/unit/models/test_dag.py index ead765b31716d..17fdac7b7ba8d 100644 --- a/airflow-core/tests/unit/models/test_dag.py +++ b/airflow-core/tests/unit/models/test_dag.py @@ -55,7 +55,6 @@ DagOwnerAttributes, DagTag, clear_team_name_cache, - get_asset_triggered_next_run_info, get_next_data_interval, get_run_data_interval, ) @@ -2585,7 +2584,12 @@ def test_dags_needing_dagruns_assets(self, dag_maker, session): # add queue records so we'll need a run dag_model = session.scalar(select(DagModel).where(DagModel.dag_id == dag.dag_id)) asset_model: AssetModel = dag_model.schedule_assets[0] - session.add(AssetDagRunQueue(asset_id=asset_model.id, target_dag_id=dag_model.dag_id)) + event = AssetEvent(asset_id=asset_model.id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + session.add( + AssetDagRunQueue(asset_id=asset_model.id, target_dag_id=dag_model.dag_id, asset_event_id=event.id) + ) session.flush() query, _ = DagModel.dags_needing_dagruns(session) dag_models = query.all() @@ -2639,7 +2643,10 @@ def test_dags_needing_dagruns_skips_adrq_when_serialized_dag_missing( session.add(dag_model) session.flush() - session.add(AssetDagRunQueue(asset_id=asset_id, target_dag_id=orphan_dag_id)) + event = AssetEvent(asset_id=asset_id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + session.add(AssetDagRunQueue(asset_id=asset_id, target_dag_id=orphan_dag_id, asset_event_id=event.id)) session.flush() with caplog.at_level(logging.DEBUG, logger="airflow.models.dag"): @@ -2707,10 +2714,14 @@ def test_dags_needing_dagruns_missing_serialized_debug_lists_sorted_dag_ids( ) session.flush() + event_z = AssetEvent(asset_id=id_z, timestamp=timezone.utcnow()) + event_a = AssetEvent(asset_id=id_a, timestamp=timezone.utcnow()) + session.add_all([event_z, event_a]) + session.flush() session.add_all( [ - AssetDagRunQueue(asset_id=id_z, target_dag_id="ghost_z"), - AssetDagRunQueue(asset_id=id_a, target_dag_id="ghost_a"), + AssetDagRunQueue(asset_id=id_z, target_dag_id="ghost_z", asset_event_id=event_z.id), + AssetDagRunQueue(asset_id=id_a, target_dag_id="ghost_a", asset_event_id=event_a.id), ] ) session.flush() @@ -2754,7 +2765,14 @@ def test_dags_needing_dagruns_query_count(self, dag_maker, session): asset_models = dag_model.schedule_assets assert len(asset_models) == num_assets for asset_model in asset_models: - session.add(AssetDagRunQueue(asset_id=asset_model.id, target_dag_id=dag_model.dag_id)) + event = AssetEvent(asset_id=asset_model.id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + session.add( + AssetDagRunQueue( + asset_id=asset_model.id, target_dag_id=dag_model.dag_id, asset_event_id=event.id + ) + ) session.flush() # Clear identity map so N+1 on adrq.asset is exposed @@ -2788,7 +2806,12 @@ def test_dags_needing_dagruns_asset_aliases(self, dag_maker, session): # add queue records so we'll need a run dag_model = dag_maker.dag_model - session.add(AssetDagRunQueue(asset_id=asset_model.id, target_dag_id=dag_model.dag_id)) + event = AssetEvent(asset_id=asset_model.id, timestamp=timezone.utcnow()) + session.add(event) + session.flush() + session.add( + AssetDagRunQueue(asset_id=asset_model.id, target_dag_id=dag_model.dag_id, asset_event_id=event.id) + ) session.flush() query, _ = DagModel.dags_needing_dagruns(session) dag_models = query.all() @@ -3046,12 +3069,24 @@ def test_dags_needing_dagruns_triggered_date_by_dag_queued_times(self, session, pass session.flush() + asset_event_ids = { + e.asset_id: e.id + for e in session.scalars( + select(AssetEvent).where(AssetEvent.asset_id.in_([asset1_id, asset2_id])) + ) + } session.add_all( [ - AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag.dag_id, created_at=DEFAULT_DATE), + AssetDagRunQueue( + asset_id=asset1_id, + target_dag_id=dag.dag_id, + asset_event_id=asset_event_ids[asset1_id], + created_at=DEFAULT_DATE, + ), AssetDagRunQueue( asset_id=asset2_id, target_dag_id=dag.dag_id, + asset_event_id=asset_event_ids[asset2_id], created_at=DEFAULT_DATE + timedelta(hours=1), ), ] @@ -3806,71 +3841,6 @@ def test__time_restriction(dag_maker, dag_date, tasks_date, catchup, restrict): assert dag._time_restriction == restrict -def test_get_asset_triggered_next_run_info(dag_maker, clear_assets): - asset1 = Asset(uri="test://asset1", name="test_asset1", group="test-group") - asset2 = Asset(uri="test://asset2", group="test-group") - asset3 = Asset(uri="test://asset3", group="test-group") - with dag_maker(dag_id="assets-1", schedule=[asset2]): - pass - dag1 = dag_maker.dag - - with dag_maker(dag_id="assets-2", schedule=[asset1, asset2]): - pass - dag2 = dag_maker.dag - - with dag_maker(dag_id="assets-3", schedule=[asset1, asset2, asset3]): - pass - dag3 = dag_maker.dag - - session = dag_maker.session - asset1_id = session.scalar(select(AssetModel.id).where(AssetModel.uri == asset1.uri)) - session.bulk_save_objects( - [ - AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag2.dag_id), - AssetDagRunQueue(asset_id=asset1_id, target_dag_id=dag3.dag_id), - ] - ) - session.flush() - - assets = session.execute(select(AssetModel.uri).order_by(AssetModel.id)).all() - - info = get_asset_triggered_next_run_info([dag1.dag_id], session=session) - assert info[dag1.dag_id] == { - "ready": 0, - "total": 1, - "uri": assets[0].uri, - } - - # This time, check both dag2 and dag3 at the same time (tests filtering) - info = get_asset_triggered_next_run_info([dag2.dag_id, dag3.dag_id], session=session) - assert info[dag2.dag_id] == { - "ready": 1, - "total": 2, - "uri": "", - } - assert info[dag3.dag_id] == { - "ready": 1, - "total": 3, - "uri": "", - } - - -@pytest.mark.need_serialized_dag -def test_get_asset_triggered_next_run_info_with_unresolved_asset_alias(dag_maker, clear_assets): - asset_alias1 = AssetAlias(name="alias") - with dag_maker(dag_id="dag-1", schedule=[asset_alias1]): - pass - dag1 = dag_maker.dag - session = dag_maker.session - session.flush() - - info = get_asset_triggered_next_run_info([dag1.dag_id], session=session) - assert info == {} - - dag1_model = DagModel.get_dagmodel(dag1.dag_id) - assert dag1_model.get_asset_triggered_next_run_info(session=session) is None - - @pytest.mark.parametrize( "run_id_type", [DagRunType.BACKFILL_JOB, DagRunType.SCHEDULED, DagRunType.ASSET_TRIGGERED], diff --git a/airflow-core/tests/unit/models/test_taskinstance.py b/airflow-core/tests/unit/models/test_taskinstance.py index 53fa99aa22460..6a348a3f2ca26 100644 --- a/airflow-core/tests/unit/models/test_taskinstance.py +++ b/airflow-core/tests/unit/models/test_taskinstance.py @@ -3877,14 +3877,15 @@ def test_runtime_partition_key_does_not_backfill_dag_run_when_none(dag_maker, se @pytest.mark.backend("sqlite") def test_runtime_partition_key_backfill_does_not_deadlock_on_sqlite(dag_maker, session): - """Regression test for the SQLite ``database is locked`` deadlock between the - writes in ``register_asset_changes_in_db`` and the second connection that - ``_create_asset_event`` used to open. - - On file-based SQLite (the default ``-b sqlite`` test backend) the two - connections compete for the same RESERVED lock; the SQLite branch of - ``_create_asset_event`` must add the event directly to the caller's session - instead of opening a second connection. + """Regression test for the SQLite ``database is locked`` deadlock. + + This happens when a second connection is used to trigger while + ``register_asset_changes_in_db`` was writing. + + The asset event is now created inline on the caller's session (see + ``AssetManager.register_asset_change``) instead of opening a side session, so + on file-based SQLite (the default ``-b sqlite`` test backend) there is no + longer a second connection competing for the same RESERVED lock. """ asset = Asset(name="hello") with dag_maker(dag_id="rt_pk_backfill_sqlite", schedule=PartitionedAtRuntime()) as dag: diff --git a/airflow-core/tests/unit/models/test_trigger.py b/airflow-core/tests/unit/models/test_trigger.py index 91a6a92e27cb3..b48f00622fe9b 100644 --- a/airflow-core/tests/unit/models/test_trigger.py +++ b/airflow-core/tests/unit/models/test_trigger.py @@ -34,6 +34,8 @@ from airflow.models import TaskInstance, Trigger from airflow.models.asset import AssetEvent, AssetModel, AssetWatcherModel from airflow.models.callback import Callback, TriggererCallback +from airflow.models.taskinstancehistory import TaskInstanceHistory +from airflow.models.trigger import handle_event_submit from airflow.models.xcom import XComModel from airflow.providers.standard.operators.empty import EmptyOperator from airflow.sdk.definitions.callback import AsyncCallback @@ -48,7 +50,7 @@ TriggerEvent, ) from airflow.utils.session import create_session -from airflow.utils.state import State +from airflow.utils.state import State, TaskInstanceState from tests_common.test_utils.asserts import assert_queries_count from tests_common.test_utils.config import conf_vars @@ -256,6 +258,72 @@ def test_submit_event_no_n_plus_one_for_assets(_, session, asset_count, expected Trigger.submit_event(trigger_id, TriggerEvent("payload"), session=session) +@pytest.mark.parametrize( + "stored_next_kwargs", + [ + # Decoding blows up: serde rejects the class name, and the BaseSerialization fallback then + # trips over the missing legacy keys. + pytest.param( + {"__classname__": "not.allowed.Thing", "__version__": 1, "__data__": {}}, + id="undecodable", + ), + # Decodes cleanly, but not into a dict: legacy encoding of a bare datetime. + pytest.param({"__type": "datetime", "__var": 1735689600.0}, id="not-a-dict"), + ], +) +def test_handle_event_submit_fails_task_with_unusable_next_kwargs( + session, create_task_instance, stored_next_kwargs +): + """ + Tests that stored kwargs which cannot be turned into a dict fail the task instance instead of + raising out of ``handle_event_submit``. Its callers walk every waiting task instance in one + pass, so one unusable payload must not abort them. + """ + task_instance = create_task_instance( + session=session, logical_date=timezone.utcnow(), state=State.DEFERRED + ) + task_instance.next_method = "execute_complete" + task_instance.next_kwargs = stored_next_kwargs + session.flush() + + handle_event_submit(TriggerEvent("payload"), task_instance=task_instance, session=session) + + session.refresh(task_instance) + assert task_instance.state == State.SCHEDULED + assert task_instance.next_method == "__fail__" + assert task_instance.trigger_id is None + assert "event" not in task_instance.next_kwargs + assert "stored next_kwargs could not be decoded" in task_instance.next_kwargs["error"] + # The traceback reaches the task log only through next_kwargs, and the runtime joins the list. + assert isinstance(task_instance.next_kwargs["traceback"], list) + + +def test_handle_event_submit_fails_task_when_the_event_payload_cannot_be_serialized( + session, create_task_instance +): + """A payload serde cannot encode is reported as such, not as unreadable stored kwargs. + + Blaming the stored kwargs would point the Dag author at database state that was never the + problem. + """ + task_instance = create_task_instance( + session=session, logical_date=timezone.utcnow(), state=State.DEFERRED + ) + task_instance.next_method = "execute_complete" + task_instance.next_kwargs = {} + session.flush() + + # serde refuses any dict carrying its reserved keys, at any depth. + handle_event_submit( + TriggerEvent({"__classname__": "anything"}), task_instance=task_instance, session=session + ) + + session.refresh(task_instance) + assert task_instance.state == State.SCHEDULED + assert task_instance.next_method == "__fail__" + assert "event payload could not be serialized" in task_instance.next_kwargs["error"] + + def test_submit_failure(session, create_task_instance): """ Tests that failures submitted to a trigger fail their dependent @@ -296,11 +364,15 @@ def test_submit_event_task_end(mock_utcnow, session, create_task_instance, event # Make a trigger trigger = Trigger(classpath="does.not.matter", kwargs={}) session.add(trigger) - # Make a TaskInstance that's deferred and waiting on it + # Make a TaskInstance that's deferred and waiting on it. A deferred task has + # already started running, so it has a start_date; set one so duration can be + # computed. Unlike set_state, handle_failure (used by the FAILED path) does not + # synthesize a missing start_date, matching the scheduler executor-event path. task_instance = create_task_instance( session=session, logical_date=timezone.utcnow(), state=State.DEFERRED ) task_instance.trigger_id = trigger.id + task_instance.start_date = now.subtract(seconds=10) session.commit() def get_xcoms(ti): @@ -362,6 +434,73 @@ def test_submit_event_task_end_callback_includes_version_data(mock_send, session assert request.version_data == version_data +@pytest.mark.parametrize( + ("retries", "expected_state", "expected_callback_type", "expect_history_row"), + [ + (1, TaskInstanceState.UP_FOR_RETRY, TaskInstanceState.UP_FOR_RETRY, True), + (0, TaskInstanceState.FAILED, TaskInstanceState.FAILED, False), + ], +) +@patch("airflow.callbacks.database_callback_sink.DatabaseCallbackSink.send") +def test_submit_event_task_end_failed_respects_retries( + mock_send, + session, + create_task_instance, + retries, + expected_state, + expected_callback_type, + expect_history_row, +): + """A trigger-emitted TaskFailedEvent should respect retry-eligibility: a deferred task with + retries remaining goes UP_FOR_RETRY (on_retry_callback), not straight to FAILED. + + On the retry path, the finished try must also be archived to task_instance_history so + prior-try log lookups keep working after the trigger ends the deferred try. + """ + trigger = Trigger(classpath="does.not.matter", kwargs={}) + session.add(trigger) + task_instance = create_task_instance( + session=session, + logical_date=timezone.utcnow(), + state=State.DEFERRED, + default_args={"retries": retries}, + ) + task_instance.trigger_id = trigger.id + task_instance.try_number = 1 + task_instance.max_tries = retries + old_ti_id = task_instance.id + session.commit() + + Trigger.submit_event(trigger.id, TaskFailedEvent(), session=session) + session.flush() + + ti = session.scalar(select(TaskInstance)) + assert ti.state == expected_state + + mock_send.assert_called_once() + request = mock_send.call_args.kwargs["callback"] + assert request.task_callback_type == expected_callback_type + + assert ti.next_method is None + assert ti.next_kwargs is None + assert ti.end_date is not None + + tih = session.scalars( + select(TaskInstanceHistory).where( + TaskInstanceHistory.dag_id == ti.dag_id, + TaskInstanceHistory.task_id == ti.task_id, + TaskInstanceHistory.run_id == ti.run_id, + ) + ).all() + if expect_history_row: + assert len(tih) == 1 + assert ti.id != old_ti_id + assert tih[0].task_instance_id == old_ti_id + else: + assert tih == [] + assert ti.id == old_ti_id + + @pytest.fixture def create_triggerer(): """Fixture factory which creates individual test Triggerer instances.""" diff --git a/airflow-core/tests/unit/serialization/test_dag_serialization.py b/airflow-core/tests/unit/serialization/test_dag_serialization.py index 4708365e846e2..7852c25dc5ee8 100644 --- a/airflow-core/tests/unit/serialization/test_dag_serialization.py +++ b/airflow-core/tests/unit/serialization/test_dag_serialization.py @@ -52,6 +52,7 @@ from airflow.dag_processing.dagbag import DagBag from airflow.exceptions import ( AirflowException, + DeserializationError, ParamValidationError, SerializationError, ) @@ -77,7 +78,7 @@ from airflow.serialization.definitions.param import SerializedParam from airflow.serialization.definitions.xcom_arg import SchedulerPlainXComArg from airflow.serialization.encoders import ensure_serialized_asset -from airflow.serialization.enums import Encoding +from airflow.serialization.enums import DagAttributeTypes, Encoding from airflow.serialization.json_schema import load_dag_schema_dict from airflow.serialization.serialized_objects import ( BaseSerialization, @@ -519,6 +520,10 @@ def timetable_plugin(monkeypatch: pytest.MonkeyPatch): ) +class PluginContributedError(AirflowException): + """Defined outside airflow.exceptions on purpose: resolution must not be limited to exceptions Airflow itself ships.""" + + class TestStringifiedDAGs: """Unit tests for stringified DAGs.""" @@ -2806,6 +2811,120 @@ def test_create_dagrun_accepts_partition_key_for_partitioned_at_runtime_dag(self dr = dag_maker.create_dagrun(partition_key="runtime-key") assert dr.partition_key == "runtime-key" + @pytest.mark.parametrize( + ("module_name", "attr_name"), + [ + pytest.param("subprocess", "check_output", id="callable_in_loaded_module"), + pytest.param("os", "system", id="another_callable"), + pytest.param("builtins", "eval", id="builtin_callable"), + pytest.param("airflow.exceptions", "NoSuchThing", id="missing_attr"), + ], + ) + def test_airflow_exc_deserialization_rejects_a_name_in_a_loaded_module(self, module_name, attr_name): + """Having the module loaded is not enough -- the name must be an AirflowException subclass. + + The module is imported by the test first, so the rejection cannot be an artefact of the + module simply being absent. + """ + importlib.import_module(module_name) + encoded = BaseSerialization._encode( + BaseSerialization.serialize( + {"exc_cls_name": f"{module_name}.{attr_name}", "args": [], "kwargs": {}} + ), + type_=DagAttributeTypes.AIRFLOW_EXC_SER, + ) + with pytest.raises(DeserializationError, match="Refusing to deserialize unknown exception class"): + BaseSerialization.deserialize(encoded) + + @pytest.mark.parametrize( + "exc_cls_name", + [ + pytest.param("not.a.loaded.module.Thing", id="unloaded_module"), + pytest.param("NoModulePart", id="no_module_part"), + pytest.param("", id="empty"), + ], + ) + def test_airflow_exc_deserialization_rejects_an_unresolvable_name(self, exc_cls_name): + encoded = BaseSerialization._encode( + BaseSerialization.serialize({"exc_cls_name": exc_cls_name, "args": [], "kwargs": {}}), + type_=DagAttributeTypes.AIRFLOW_EXC_SER, + ) + with pytest.raises(DeserializationError, match="Refusing to deserialize unknown exception class"): + BaseSerialization.deserialize(encoded) + + def test_airflow_exc_deserialization_does_not_import_the_named_module(self): + """Resolution reads ``sys.modules``; it never imports what the blob names.""" + module_name = "airflow_exc_module_that_must_not_be_imported" + encoded = BaseSerialization._encode( + BaseSerialization.serialize({"exc_cls_name": f"{module_name}.Boom", "args": [], "kwargs": {}}), + type_=DagAttributeTypes.AIRFLOW_EXC_SER, + ) + with mock.patch.object( + importlib, "import_module", side_effect=AssertionError("imported"), autospec=True + ): + with pytest.raises(DeserializationError): + BaseSerialization.deserialize(encoded) + assert module_name not in sys.modules + + def test_airflow_exc_deserialization_roundtrips_airflow_exception(self): + result = BaseSerialization.deserialize(BaseSerialization.serialize(AirflowException("boom"))) + assert isinstance(result, AirflowException) + assert result.args == ("boom",) + + def test_airflow_exc_deserialization_resolves_a_subclass_outside_airflow(self): + """Resolves as long as its module is loaded, registration order doesn't matter.""" + result = BaseSerialization.deserialize(BaseSerialization.serialize(PluginContributedError("boom"))) + assert isinstance(result, PluginContributedError) + assert result.args == ("boom",) + + @pytest.mark.parametrize( + "exc_cls_name", + [ + "airflow.exceptions.AirflowException", + "airflow.exceptions.AirflowNotFoundException", + "airflow.exceptions.ParamValidationError", + "airflow.sdk.exceptions.AirflowException", + ], + ) + def test_airflow_exc_deserialization_accepts_the_pre_3_2_module_spelling(self, exc_cls_name): + """A blob written before these exceptions moved to ``airflow.sdk.exceptions`` still reads. + + 3.0/3.1 stored ``airflow.exceptions.``; 3.2.0 moved the classes and left a re-export + behind. Both spellings have to resolve, or upgrading strands every stored blob that carries + an exception node. + """ + encoded = BaseSerialization._encode( + BaseSerialization.serialize({"exc_cls_name": exc_cls_name, "args": ["boom"], "kwargs": {}}), + type_=DagAttributeTypes.AIRFLOW_EXC_SER, + ) + result = BaseSerialization.deserialize(encoded) + assert isinstance(result, AirflowException) + + def test_base_exc_deserialization_rejects_non_allowlisted_builtin(self): + """A BASE_EXC_SER name outside the {KeyError, AttributeError} the encoder emits is rejected.""" + # ``eval`` is the weaponisable case; ``ValueError`` is a harmless builtin the encoder + # never emits as BASE_EXC_SER -- both must be rejected. + for name in ("eval", "ValueError"): + encoded = BaseSerialization._encode( + BaseSerialization.serialize({"exc_cls_name": name, "args": ["1"], "kwargs": {}}), + type_=DagAttributeTypes.BASE_EXC_SER, + ) + with pytest.raises( + DeserializationError, match="Refusing to deserialize unsupported builtin exception" + ): + BaseSerialization.deserialize(encoded) + + @pytest.mark.parametrize("exc_type", [KeyError, AttributeError]) + def test_base_exc_serialize_deserialize_round_trip(self, exc_type): + """Pins the allow-list to the encode branch, by going through ``serialize`` rather than + hand-building the node: if that branch ever accepts another builtin, this fails.""" + result = BaseSerialization.deserialize(BaseSerialization.serialize(exc_type("boom"))) + + assert isinstance(result, exc_type) + # The encode branch stores ``[var.args]``, so the args arrive nested. Pre-existing shape, + # asserted as it is rather than as it ought to be. + assert result.args == (("boom",),) + def test_kubernetes_optional(): """Test that serialization module loads without kubernetes, but deserialization of PODs requires it""" diff --git a/airflow-core/tests/unit/timetables/test_assets_timetable.py b/airflow-core/tests/unit/timetables/test_assets_timetable.py index 3c12c886f283f..ccdc395ae410d 100644 --- a/airflow-core/tests/unit/timetables/test_assets_timetable.py +++ b/airflow-core/tests/unit/timetables/test_assets_timetable.py @@ -295,7 +295,10 @@ def test_asset_dag_run_queue_processing(self, session, dag_maker, create_test_as # Add AssetDagRunQueue entries to simulate asset event processing for am in asset_models: - session.add(AssetDagRunQueue(asset_id=am.id, target_dag_id=dag.dag_id)) + event = AssetEvent(asset_id=am.id) + session.add(event) + session.flush() + session.add(AssetDagRunQueue(asset_id=am.id, target_dag_id=dag.dag_id, asset_event_id=event.id)) session.commit() # Fetch and evaluate asset triggers for all DAGs affected by asset events diff --git a/airflow-core/tests/unit/utils/test_event_scheduler.py b/airflow-core/tests/unit/utils/test_event_scheduler.py index 641d8dd0f909c..5a9e68b270ef0 100644 --- a/airflow-core/tests/unit/utils/test_event_scheduler.py +++ b/airflow-core/tests/unit/utils/test_event_scheduler.py @@ -19,6 +19,8 @@ from unittest import mock +import pytest + from airflow.utils.event_scheduler import EventScheduler @@ -38,3 +40,34 @@ def test_call_regular_interval(self): assert len(timers.queue) == 2 somefunction.assert_called_once() assert timers.queue[0].time < timers.queue[1].time + + def test_call_regular_interval_propagates_exception_by_default(self): + """Without opting in, an action's exception still propagates (unchanged default behavior).""" + failing_action = mock.MagicMock(side_effect=RuntimeError("boom")) + + timers = EventScheduler() + timers.call_regular_interval(30, failing_action) + assert len(timers.queue) == 1 + + with pytest.raises(RuntimeError, match="boom"): + timers.queue[0].action() + + failing_action.assert_called_once() + # The next cycle was never scheduled because the exception propagated. + assert len(timers.queue) == 1 + + def test_call_regular_interval_non_fatal_swallows_action_exception(self): + """With non_fatal=True, a raising action is swallowed and the next cycle is still scheduled.""" + failing_action = mock.MagicMock(side_effect=RuntimeError("boom")) + + timers = EventScheduler() + timers.call_regular_interval(30, failing_action, non_fatal=True) + assert len(timers.queue) == 1 + + # Should not raise, even though the action does. + timers.queue[0].action() + + failing_action.assert_called_once() + # The next cycle was still scheduled despite the exception. + assert len(timers.queue) == 2 + assert timers.queue[0].time < timers.queue[1].time diff --git a/airflow-ctl/docs/images/command_hashes.txt b/airflow-ctl/docs/images/command_hashes.txt index 65c22154bc3c0..4e48ad37fee6f 100644 --- a/airflow-ctl/docs/images/command_hashes.txt +++ b/airflow-ctl/docs/images/command_hashes.txt @@ -1,12 +1,12 @@ main:2c358f5fc894541cab11854dcd67658a -assets:6419e20452692f577c4c6f570b74be0c +assets:f225579fbecb3d919d152b4f1ba83185 auth:d79e9c7d00c432bdbcbc2a86e2e32053 backfill:74c8737b0a62a86ed3605fa9e6165874 config:a3d936cb15fe3b547bf6c82cf93d923f connections:942f9f88cb908c28bf5c19159fc5065b dags:ea477b70a1c3bcf8ed603cdbd11e5ab6 dagrun:07035226eaaff0a557d3ad9aab4b41b5 -jobs:a5b644c5da8889443bb40ee10b599270 +jobs:d4af478f28dae48ee18d43b998edc345 pools:19efe105b9515ab1926ebcaf0e028d71 providers:34502fe09dc0b8b0a13e7e46efdffda6 taskinstances:bea84117114c2438eb7e7026f6bb7042 diff --git a/airflow-ctl/docs/images/output_assets.svg b/airflow-ctl/docs/images/output_assets.svg index 12214b4fd7441..c5dc475c84a44 100644 --- a/airflow-ctl/docs/images/output_assets.svg +++ b/airflow-ctl/docs/images/output_assets.svg @@ -19,108 +19,108 @@ font-weight: 700; } - .terminal-2929493646-matrix { + .terminal-2396817043-matrix { font-family: Fira Code, monospace; font-size: 20px; line-height: 24.4px; font-variant-east-asian: full-width; } - .terminal-2929493646-title { + .terminal-2396817043-title { font-size: 18px; font-weight: bold; font-family: arial; } - .terminal-2929493646-r1 { fill: #ff8700 } -.terminal-2929493646-r2 { fill: #c5c8c6 } -.terminal-2929493646-r3 { fill: #808080 } -.terminal-2929493646-r4 { fill: #68a0b3 } + .terminal-2396817043-r1 { fill: #ff8700 } +.terminal-2396817043-r2 { fill: #c5c8c6 } +.terminal-2396817043-r3 { fill: #808080 } +.terminal-2396817043-r4 { fill: #68a0b3 } - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -132,37 +132,37 @@ - + - - Usage:airflowctl assets [-h] COMMAND... - -Perform Assets operations - -Positional Arguments: -COMMAND -create-eventCreate an event for a given asset -delete-dag-queued-events -Delete all queued asset events for a given Dag -delete-queued-event -Delete a specific queued asset event for a given  -Dag and asset -delete-queued-events -Delete all queued events for a given asset -getRetrieve an asset by its ID -get-by-aliasRetrieve an asset by its alias -get-dag-queued-event -Retrieve a specific queued asset event for a given  -Dag and asset -get-dag-queued-events -List queued asset events for a given Dag -get-queued-eventsList queued events for a given asset -listList all assets -list-aliasesList all asset aliases -materializeTrigger materialization of an asset by its ID - -Options: --h, --helpshow this help message and exit + + Usage:airflowctl assets [-h] COMMAND... + +Perform Assets operations + +Positional Arguments: +COMMAND +create-eventCreate an event for a given asset +delete-dag-queued-events +Delete all queued asset events for a given Dag +delete-queued-event +Delete a specific queued asset event for a given  +Dag and asset +delete-queued-events +Delete all queued events for a given asset +getRetrieve an asset by its ID +get-aliasRetrieve an asset alias by its ID +get-dag-queued-event +Retrieve a specific queued asset event for a given  +Dag and asset +get-dag-queued-events +List queued asset events for a given Dag +get-queued-eventsList queued events for a given asset +listList all assets +list-aliasesList all asset aliases +materializeTrigger materialization of an asset by its ID + +Options: +-h, --helpshow this help message and exit diff --git a/airflow-ctl/docs/images/output_jobs.svg b/airflow-ctl/docs/images/output_jobs.svg index 13b31d2caedc4..9ff6f8753c162 100644 --- a/airflow-ctl/docs/images/output_jobs.svg +++ b/airflow-ctl/docs/images/output_jobs.svg @@ -1,4 +1,4 @@ - + - - + + - + - + - + - + - + - + - + - + - + + + + - + - + - - Usage:airflowctl jobs [-h] COMMAND... - -Perform Jobs operations - -Positional Arguments: -COMMAND -listList scheduler, triggerer, and other Airflow jobs - -Options: --h, --helpshow this help message and exit + + Usage:airflowctl jobs [-h] COMMAND... + +Perform Jobs operations + +Positional Arguments: +COMMAND +checkCheck if job(s) are still alive. +listList scheduler, triggerer, and other Airflow jobs + +Options: +-h, --helpshow this help message and exit diff --git a/airflow-ctl/src/airflowctl/api/operations.py b/airflow-ctl/src/airflowctl/api/operations.py index 94b31010abbc2..4916203247a5d 100644 --- a/airflow-ctl/src/airflowctl/api/operations.py +++ b/airflow-ctl/src/airflowctl/api/operations.py @@ -220,7 +220,7 @@ def safe_validate(content: bytes) -> BaseModel: raw = fill_missing_fields(json.loads(content), data_model) return data_model.model_validate(raw) # type: ignore[union-attr] - self.response = self.client.get(path, params=shared_params) + self.response = self.client.get(path, params={**shared_params, "offset": offset}) first_pass = safe_validate(self.response.content) total_entries = first_pass.total_entries # type: ignore[attr-defined] if total_entries < limit: @@ -264,9 +264,9 @@ def get(self, asset_id: str) -> AssetResponse | ServerResponseError: self.response = self.client.get(f"assets/{asset_id}") return AssetResponse.model_validate_json(self.response.content) - def get_by_alias(self, alias: str) -> AssetAliasResponse | ServerResponseError: - """Get an asset by alias from the API server.""" - self.response = self.client.get(f"assets/aliases/{alias}") + def get_alias(self, asset_alias_id: str) -> AssetAliasResponse | ServerResponseError: + """Get an asset alias by its ID from the API server.""" + self.response = self.client.get(f"assets/aliases/{asset_alias_id}") return AssetAliasResponse.model_validate_json(self.response.content) def list(self) -> AssetCollectionResponse | ServerResponseError: diff --git a/airflow-ctl/src/airflowctl/ctl/cli_config.py b/airflow-ctl/src/airflowctl/ctl/cli_config.py index 787a53ce074b6..bbf5e59f8e9d9 100755 --- a/airflow-ctl/src/airflowctl/ctl/cli_config.py +++ b/airflow-ctl/src/airflowctl/ctl/cli_config.py @@ -354,6 +354,35 @@ def _load_help_texts_yaml() -> dict[str, dict[str, str]]: choices=("overwrite", "fail", "skip"), ) +# Jobs command args +ARG_JOB_TYPE_FILTER = Arg( + flags=("--job-type",), + choices=("SchedulerJob", "TriggererJob", "DagProcessorJob"), + help="The type of job(s) that will be checked.", +) +ARG_JOB_HOSTNAME_FILTER = Arg( + flags=("--hostname",), + type=str, + default=None, + help="The hostname of job(s) that will be checked.", +) +ARG_JOB_LOCAL_FILTER = Arg( + flags=("--local",), + action="store_true", + help="If passed, this command will only show jobs from the local host.", +) +ARG_JOB_LIMIT = Arg( + flags=("--limit",), + type=positive_int(allow_zero=True), + default=1, + help="The number of recent jobs that will be checked. To disable limit, set 0.", +) +ARG_JOB_ALLOW_MULTIPLE = Arg( + flags=("--allow-multiple",), + action="store_true", + help="If passed, this command will be successful even if multiple matching alive jobs are found.", +) + # Config arguments ARG_CONFIG_SECTION = Arg( flags=("--section",), @@ -1185,6 +1214,21 @@ def merge_commands( ), ) +JOB_COMMANDS = ( + ActionCommand( + name="check", + help="Check if job(s) are still alive.", + func=lazy_load_command("airflowctl.ctl.commands.jobs_command.check"), + args=( + ARG_JOB_TYPE_FILTER, + ARG_JOB_HOSTNAME_FILTER, + ARG_JOB_LOCAL_FILTER, + ARG_JOB_LIMIT, + ARG_JOB_ALLOW_MULTIPLE, + ), + ), +) + core_commands: list[CLICommand] = [ GroupCommand( name="auth", @@ -1207,6 +1251,11 @@ def merge_commands( help="Manage Airflow Dags", subcommands=DAG_COMMANDS, ), + GroupCommand( + name="jobs", + help="Manage Airflow jobs", + subcommands=JOB_COMMANDS, + ), GroupCommand( name="pools", help="Manage Airflow pools", diff --git a/airflow-ctl/src/airflowctl/ctl/commands/jobs_command.py b/airflow-ctl/src/airflowctl/ctl/commands/jobs_command.py new file mode 100644 index 0000000000000..0eb9a47a62a66 --- /dev/null +++ b/airflow-ctl/src/airflowctl/ctl/commands/jobs_command.py @@ -0,0 +1,50 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import socket + +import rich + +from airflowctl.api.client import NEW_API_CLIENT, ClientKind, provide_api_client + + +@provide_api_client(kind=ClientKind.CLI) +def check(args, api_client=NEW_API_CLIENT) -> None: + """Check if job(s) are still alive.""" + if args.allow_multiple and args.limit == 1: + raise SystemExit( + "To use option --allow-multiple, you must set the limit to a value greater than 1 " + "or 0 to disable it." + ) + if args.hostname and args.local: + raise SystemExit("You can't use --hostname and --local at the same time.") + + hostname = socket.getfqdn() if args.local else args.hostname + alive_jobs = api_client.jobs.list(job_type=args.job_type, hostname=hostname, is_alive=True).jobs + if args.limit > 0: + alive_jobs = alive_jobs[: args.limit] + + count_alive_jobs = len(alive_jobs) + if count_alive_jobs == 0: + raise SystemExit("No alive jobs found.") + if count_alive_jobs > 1 and not args.allow_multiple: + raise SystemExit(f"Found {count_alive_jobs} alive jobs. Expected only one.") + if count_alive_jobs == 1: + rich.print("Found one alive job.") + else: + rich.print(f"Found {count_alive_jobs} alive jobs.") diff --git a/airflow-ctl/src/airflowctl/ctl/help_texts.yaml b/airflow-ctl/src/airflowctl/ctl/help_texts.yaml index 08626e0093096..4944c60dbdf8b 100644 --- a/airflow-ctl/src/airflowctl/ctl/help_texts.yaml +++ b/airflow-ctl/src/airflowctl/ctl/help_texts.yaml @@ -17,7 +17,7 @@ --- assets: get: "Retrieve an asset by its ID" - get-by-alias: "Retrieve an asset by its alias" + get-alias: "Retrieve an asset alias by its ID" list: "List all assets" list-aliases: "List all asset aliases" create-event: "Create an event for a given asset" diff --git a/airflow-ctl/tests/airflow_ctl/api/test_operations.py b/airflow-ctl/tests/airflow_ctl/api/test_operations.py index 4868841bc2fd2..52fa594afd5ee 100644 --- a/airflow-ctl/tests/airflow_ctl/api/test_operations.py +++ b/airflow-ctl/tests/airflow_ctl/api/test_operations.py @@ -86,6 +86,7 @@ PluginResponse, PoolBody, PoolCollectionResponse, + PoolPatchBody, PoolResponse, ProviderCollectionResponse, ProviderResponse, @@ -250,6 +251,29 @@ def test_execute_list_sends_limit_on_subsequent_pages(self): for call in mock_client.get.call_args_list: assert call.kwargs["params"]["limit"] == 2 + def test_execute_list_sends_offset_to_server(self): + """``offset`` must reach the server on the first request, not only on subsequent pages.""" + rows = [{"name": name} for name in "abcdef"] + + def paged(path, params): + # An absent ``offset`` is what the server would see as its own default of 0. + start = params.get("offset", 0) + return Mock( + content=json.dumps( + {"hellos": rows[start : start + params["limit"]], "total_entries": len(rows)} + ) + ) + + mock_client = Mock() + mock_client.get.side_effect = paged + base_operation = BaseOperations(client=mock_client) + + response = base_operation.execute_list( + path="hello", data_model=HelloCollectionResponse, offset=2, limit=2 + ) + + assert [hello.name for hello in response.hellos] == ["c", "d", "e", "f"] + @pytest.mark.parametrize("limit", [0, -1]) def test_execute_list_rejects_non_positive_limit(self, limit): mock_client = Mock() @@ -263,6 +287,7 @@ def test_execute_list_rejects_non_positive_limit(self, limit): class TestAssetsOperations: asset_id: int = 1 + asset_alias_id: int = 2 dag_id: str = "dag_id" before: str = "2024-12-31T23:59:59+00:00" asset_response = AssetResponse( @@ -280,7 +305,7 @@ class TestAssetsOperations: group="group", ) asset_alias_response = AssetAliasResponse( - id=asset_id, + id=asset_alias_id, name="asset", group="group", ) @@ -371,13 +396,13 @@ def handle_request(request: httpx.Request) -> httpx.Response: response = client.assets.get(self.asset_id) assert response == self.asset_response - def test_get_by_alias(self): + def test_get_alias(self): def handle_request(request: httpx.Request) -> httpx.Response: - assert request.url.path == f"/api/v2/assets/aliases/{self.asset_id}" + assert request.url.path == f"/api/v2/assets/aliases/{self.asset_alias_id}" return httpx.Response(200, json=json.loads(self.asset_alias_response.model_dump_json())) client = make_api_client(transport=httpx.MockTransport(handle_request)) - response = client.assets.get_by_alias(self.asset_id) + response = client.assets.get_alias(self.asset_alias_id) assert response == self.asset_alias_response def test_list(self): @@ -808,6 +833,16 @@ def handle_request(request: httpx.Request) -> httpx.Response: response = client.connections.bulk(connections=connection_bulk_body) assert response == self.connection_bulk_response + def test_create_defaults(self): + def handle_request(request: httpx.Request) -> httpx.Response: + assert request.url.path == "/api/v2/connections/defaults" + assert request.method == "POST" + return httpx.Response(200, json={}) + + client = make_api_client(transport=httpx.MockTransport(handle_request)) + response = client.connections.create_defaults() + assert response is None + def test_delete(self): def handle_request(request: httpx.Request) -> httpx.Response: assert request.url.path == f"/api/v2/connections/{self.connection_id}" @@ -1613,6 +1648,7 @@ class TestPoolsOperations: pools=[pool_response], total_entries=1, ) + pool_patch_body = PoolPatchBody(pool=pool_name, description="description") pool_bulk_response = BulkResponse( create=BulkActionResponse(success=[pool_name], errors=[]), update=None, @@ -1664,6 +1700,16 @@ def handle_request(request: httpx.Request) -> httpx.Response: response = client.pools.delete(self.pool_name) assert response == self.pool_name + def test_update(self): + def handle_request(request: httpx.Request) -> httpx.Response: + assert request.url.path == f"/api/v2/pools/{self.pool_name}" + assert request.method == "PATCH" + return httpx.Response(200, json=json.loads(self.pool_response.model_dump_json())) + + client = make_api_client(transport=httpx.MockTransport(handle_request)) + response = client.pools.update(pool_body=self.pool_patch_body) + assert response == self.pool_response + class TestProvidersOperations: provider_response = ProviderResponse( diff --git a/airflow-ctl/tests/airflow_ctl/ctl/commands/test_jobs_command.py b/airflow-ctl/tests/airflow_ctl/ctl/commands/test_jobs_command.py new file mode 100644 index 0000000000000..697faa682c4a7 --- /dev/null +++ b/airflow-ctl/tests/airflow_ctl/ctl/commands/test_jobs_command.py @@ -0,0 +1,115 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from types import SimpleNamespace +from unittest import mock + +import pytest + +from airflowctl.ctl import cli_parser +from airflowctl.ctl.commands import jobs_command + + +def _make_jobs(count: int) -> SimpleNamespace: + """Build a stand-in for the ``JobCollectionResponse`` of ``count`` already-alive jobs.""" + return SimpleNamespace(jobs=[SimpleNamespace(id=i) for i in range(count)]) + + +class TestCliJobsCheck: + parser = cli_parser.get_parser() + + @pytest.mark.parametrize( + ("argv", "alive_count", "expected_output"), + [ + (["jobs", "check", "--job-type", "SchedulerJob"], 1, "Found one alive job."), + (["jobs", "check", "--limit", "100", "--allow-multiple"], 3, "Found 3 alive jobs."), + (["jobs", "check", "--limit", "0", "--allow-multiple"], 3, "Found 3 alive jobs."), + (["jobs", "check", "--limit", "1"], 5, "Found one alive job."), + ], + ) + def test_reports_alive_jobs(self, capsys, argv, alive_count, expected_output): + api_client = mock.MagicMock() + api_client.jobs.list.return_value = _make_jobs(alive_count) + + jobs_command.check(self.parser.parse_args(argv), api_client=api_client) + + assert expected_output in capsys.readouterr().out + + @pytest.mark.parametrize( + ("argv", "local_hostname", "expected_kwargs"), + [ + ( + ["jobs", "check", "--job-type", "SchedulerJob"], + None, + {"job_type": "SchedulerJob", "hostname": None, "is_alive": True}, + ), + ( + ["jobs", "check", "--hostname", "HOSTNAME"], + None, + {"job_type": None, "hostname": "HOSTNAME", "is_alive": True}, + ), + ( + ["jobs", "check", "--local"], + "local-host", + {"job_type": None, "hostname": "local-host", "is_alive": True}, + ), + ], + ) + def test_forwards_filters_to_jobs_list(self, argv, local_hostname, expected_kwargs): + api_client = mock.MagicMock() + api_client.jobs.list.return_value = _make_jobs(1) + + with mock.patch("airflowctl.ctl.commands.jobs_command.socket.getfqdn", return_value=local_hostname): + jobs_command.check(self.parser.parse_args(argv), api_client=api_client) + + api_client.jobs.list.assert_called_once_with(**expected_kwargs) + + @pytest.mark.parametrize( + ("argv", "alive_count", "expected_error"), + [ + (["jobs", "check"], 0, r"No alive jobs found."), + (["jobs", "check", "--limit", "100"], 3, r"Found 3 alive jobs. Expected only one."), + ], + ) + def test_exits_when_alive_count_unexpected(self, argv, alive_count, expected_error): + api_client = mock.MagicMock() + api_client.jobs.list.return_value = _make_jobs(alive_count) + + with pytest.raises(SystemExit, match=expected_error): + jobs_command.check(self.parser.parse_args(argv), api_client=api_client) + + @pytest.mark.parametrize( + ("argv", "expected_error"), + [ + ( + ["jobs", "check", "--allow-multiple"], + r"To use option --allow-multiple, you must set the limit to a value greater than 1 " + r"or 0 to disable it.", + ), + ( + ["jobs", "check", "--hostname", "h", "--local"], + r"You can't use --hostname and --local at the same time", + ), + ], + ) + def test_rejects_invalid_argument_combinations(self, argv, expected_error): + api_client = mock.MagicMock() + + with pytest.raises(SystemExit, match=expected_error): + jobs_command.check(self.parser.parse_args(argv), api_client=api_client) + api_client.jobs.list.assert_not_called() diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml index 1a1460293bcea..54625dd01c442 100644 --- a/chart/templates/_helpers.yaml +++ b/chart/templates/_helpers.yaml @@ -91,13 +91,6 @@ If release name contains chart name it will be used as a full name. name: {{ template "api_secret_key_secret" . }} key: api-secret-key {{- end }} - {{- if and .IncludeJwtSecret .Values.enableBuiltInSecretEnvVars.AIRFLOW__API_AUTH__JWT_SECRET }} - - name: AIRFLOW__API_AUTH__JWT_SECRET - valueFrom: - secretKeyRef: - name: {{ template "jwt_secret" . }} - key: jwt-secret - {{- end }} {{- if contains "CeleryExecutor" .Values.executor }} {{- if and .Values.enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND (or .Values.data.resultBackendSecretName .Values.data.resultBackendConnection) }} - name: AIRFLOW__CELERY__RESULT_BACKEND @@ -148,6 +141,17 @@ If release name contains chart name it will be used as a full name. {{- end }} {{- end }} +{{/* JWT signing and validation secret, only needed by the API server and the scheduler */}} +{{- define "jwt_secret_environment" }} + {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__API_AUTH__JWT_SECRET }} + - name: AIRFLOW__API_AUTH__JWT_SECRET + valueFrom: + secretKeyRef: + name: {{ template "jwt_secret" . }} + key: jwt-secret + {{- end }} +{{- end }} + {{/* KEDA scaler database connection for the worker and triggerer autoscalers */}} {{- define "keda_airflow_environment" }} {{- if or (eq .Values.data.metadataConnection.protocol "mysql") (and .Values.pgbouncer.enabled (not .UsePgbouncer)) }} diff --git a/chart/templates/api-server/api-server-deployment.yaml b/chart/templates/api-server/api-server-deployment.yaml index bc02135ae2b04..5d6eeb3d5075b 100644 --- a/chart/templates/api-server/api-server-deployment.yaml +++ b/chart/templates/api-server/api-server-deployment.yaml @@ -151,7 +151,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if .Values.apiServer.waitForMigrations.env }} {{- tpl (toYaml .Values.apiServer.waitForMigrations.env) $ | nindent 12 }} {{- end }} @@ -225,7 +225,8 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" true) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} + {{- include "jwt_secret_environment" . | indent 10 }} {{- include "container_extra_envs" (list . .Values.apiServer.env) | indent 10 }} {{- if .Values.apiServer.extraContainers }} {{- tpl (toYaml .Values.apiServer.extraContainers) . | nindent 8 }} diff --git a/chart/templates/dag-processor/dag-processor-deployment.yaml b/chart/templates/dag-processor/dag-processor-deployment.yaml index d3596527c7765..6a3865fc366c8 100644 --- a/chart/templates/dag-processor/dag-processor-deployment.yaml +++ b/chart/templates/dag-processor/dag-processor-deployment.yaml @@ -132,7 +132,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if .Values.dagProcessor.waitForMigrations.env }} {{- tpl (toYaml .Values.dagProcessor.waitForMigrations.env) $ | nindent 12 }} {{- end }} @@ -177,7 +177,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- include "container_extra_envs" (list . .Values.dagProcessor.env) | indent 10 }} livenessProbe: initialDelaySeconds: {{ .Values.dagProcessor.livenessProbe.initialDelaySeconds }} diff --git a/chart/templates/scheduler/scheduler-deployment.yaml b/chart/templates/scheduler/scheduler-deployment.yaml index 35749fdbc5ba2..57c60f917c014 100644 --- a/chart/templates/scheduler/scheduler-deployment.yaml +++ b/chart/templates/scheduler/scheduler-deployment.yaml @@ -164,7 +164,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if .Values.scheduler.waitForMigrations.env }} {{- tpl (toYaml .Values.scheduler.waitForMigrations.env) $ | nindent 12 }} {{- end }} @@ -192,7 +192,8 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" true) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} + {{- include "jwt_secret_environment" . | indent 10 }} {{- include "container_extra_envs" (list . .Values.scheduler.env) | indent 10 }} livenessProbe: initialDelaySeconds: {{ .Values.scheduler.livenessProbe.initialDelaySeconds }} diff --git a/chart/templates/triggerer/triggerer-deployment.yaml b/chart/templates/triggerer/triggerer-deployment.yaml index 1c773d349ea9e..806a7a5df3176 100644 --- a/chart/templates/triggerer/triggerer-deployment.yaml +++ b/chart/templates/triggerer/triggerer-deployment.yaml @@ -155,7 +155,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if .Values.triggerer.waitForMigrations.env }} {{- tpl (toYaml .Values.triggerer.waitForMigrations.env) $ | nindent 12 }} {{- end }} @@ -200,7 +200,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if $keda }} {{- include "keda_airflow_environment" (merge (dict "UsePgbouncer" .Values.triggerer.keda.usePgbouncer) .) | indent 10 }} {{- end }} diff --git a/chart/templates/workers/worker-deployment.yaml b/chart/templates/workers/worker-deployment.yaml index 294b0d4101020..536e73a216a14 100644 --- a/chart/templates/workers/worker-deployment.yaml +++ b/chart/templates/workers/worker-deployment.yaml @@ -217,7 +217,7 @@ spec: - name: KRB5CCNAME value: {{ include "kerberos_ccache_path" . | quote }} {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- end }} {{- if .Values.workers.celery.waitForMigrations.enabled }} - name: wait-for-airflow-migrations @@ -242,7 +242,7 @@ spec: envFrom: {{- include "custom_airflow_environment_from" . | default "\n []" | indent 10 }} env: {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if .Values.workers.celery.waitForMigrations.env }} {{- tpl (toYaml .Values.workers.celery.waitForMigrations.env) $ | nindent 12 }} {{- end }} @@ -325,7 +325,7 @@ spec: - name: DUMB_INIT_SETSID value: "0" {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- if $keda }} {{- include "keda_airflow_environment" (merge (dict "UsePgbouncer" .Values.workers.celery.keda.usePgbouncer) .) | indent 10 }} {{- end }} @@ -434,7 +434,7 @@ spec: - name: KRB5CCNAME value: {{ include "kerberos_ccache_path" . | quote }} {{- include "custom_airflow_environment" . | indent 10 }} - {{- include "standard_airflow_environment" (merge (dict "IncludeJwtSecret" false) .) | indent 10 }} + {{- include "standard_airflow_environment" . | indent 10 }} {{- end }} {{- if .Values.workers.celery.extraContainers }} {{- tpl (toYaml .Values.workers.celery.extraContainers) . | nindent 8 }} diff --git a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py index ff82d4a29ca74..0a5b15562797c 100644 --- a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py +++ b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py @@ -376,8 +376,8 @@ def test_have_all_variables(self): "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN", "AIRFLOW_CONN_AIRFLOW_DB", "AIRFLOW__API__SECRET_KEY", - "AIRFLOW__API_AUTH__JWT_SECRET", "AIRFLOW__CELERY__BROKER_URL", + "AIRFLOW__API_AUTH__JWT_SECRET", ] expected_vars_no_jwt = [ "AIRFLOW_HOME", @@ -396,6 +396,65 @@ def test_have_all_variables(self): f"Wrong vars in {component}" ) + def test_jwt_secret_injected_into_api_server_and_scheduler(self): + docs = render_chart( + show_only=[ + "templates/api-server/api-server-deployment.yaml", + "templates/scheduler/scheduler-deployment.yaml", + ], + ) + + for doc in docs: + component = doc["metadata"]["labels"]["component"] + env_names = jmespath.search( + f"spec.template.spec.containers[?name=='{component}'].env[].name", doc + ) + assert env_names.count("AIRFLOW__API_AUTH__JWT_SECRET") == 1, ( + f"JWT secret missing from {component}" + ) + + # it must not leak into the sidecars or init containers of those same pods + other_env_names = jmespath.search( + f"[spec.template.spec.containers[?name!='{component}'], " + "spec.template.spec.initContainers][][].env[].name", + doc, + ) + assert "AIRFLOW__API_AUTH__JWT_SECRET" not in other_env_names, ( + f"JWT secret leaked into a non-main container of {component}" + ) + + def test_jwt_secret_not_injected_into_other_components(self): + docs = render_chart( + show_only=[ + "templates/workers/worker-deployment.yaml", + "templates/triggerer/triggerer-deployment.yaml", + "templates/dag-processor/dag-processor-deployment.yaml", + ], + ) + + for doc in docs: + component = doc["metadata"]["labels"]["component"] + env_names = jmespath.search( + "[spec.template.spec.containers, spec.template.spec.initContainers][][].env[].name", doc + ) + assert "AIRFLOW__API_AUTH__JWT_SECRET" not in env_names, f"JWT secret leaked into {component}" + + def test_jwt_secret_can_be_disabled(self): + docs = render_chart( + values={"enableBuiltInSecretEnvVars": {"AIRFLOW__API_AUTH__JWT_SECRET": False}}, + show_only=[ + "templates/api-server/api-server-deployment.yaml", + "templates/scheduler/scheduler-deployment.yaml", + ], + ) + + for doc in docs: + component = doc["metadata"]["labels"]["component"] + env_names = jmespath.search( + f"spec.template.spec.containers[?name=='{component}'].env[].name", doc + ) + assert "AIRFLOW__API_AUTH__JWT_SECRET" not in env_names, f"Wrong vars in {component}" + def test_have_all_config_mounts_on_init_containers(self): docs = render_chart( show_only=[ diff --git a/contributing-docs/08_static_code_checks.rst b/contributing-docs/08_static_code_checks.rst index f083a957a9aea..161bd7902b881 100644 --- a/contributing-docs/08_static_code_checks.rst +++ b/contributing-docs/08_static_code_checks.rst @@ -234,7 +234,7 @@ To check other usage types of the pre-commit framework, see `Pre-commit website - "- **Lint with ruff only:** `prek run ruff --from-ref `" - "- **Format with ruff only:** `prek run ruff-format --from-ref `" - "- **Run regular (fast) static checks:** `prek run --from-ref --stage pre-commit`" - - "- **Run manual (slower) checks:** `prek run --from-ref --stage manual`" + - "- **Run manual (slower) checks:** `prek run --from-ref --stage manual --skip compile-ui-assets-dev --skip view-skill-eval` (the skipped hooks start long-running local servers rather than checks that complete)" - "- **Build docs:** `breeze build-docs`" - "- **Determine which tests to run based on changed files:** `breeze ci selective-check --commit-ref `" .. AGENT-SKILL-END diff --git a/contributing-docs/23_provider_hook_migration_to_yaml.rst b/contributing-docs/23_provider_hook_migration_to_yaml.rst index f751f9675a901..54706f6c4dc59 100644 --- a/contributing-docs/23_provider_hook_migration_to_yaml.rst +++ b/contributing-docs/23_provider_hook_migration_to_yaml.rst @@ -60,6 +60,32 @@ Customizations for standard connection fields: placeholders: port: '5432' +external-services +~~~~~~~~~~~~~~~~~ + +List of upstream provider or model-serving services that this connection type +reaches over the network (e.g. ``OpenAI``, ``AWS Bedrock``, ``Ollama``). The registry +renders this list as a table on the provider's version page, so someone browsing the +registry can see at a glance which external services a given connection type talks +to. Because some hooks resolve the destination from a caller-supplied model +identifier (for example ``PydanticAIHook``), this list is representative, not +exhaustive -- treat it as example services, not a compatibility matrix. + +This is unrelated to the top-level ``integrations`` key in ``provider.yaml``, which +describes *framework* integrations (e.g. Kubernetes, Docker) and drives the +provider's docs pages, logo, and tags. ``external-services`` only applies inside a +``connection-types`` entry and only lists services reached over the network — it has +no effect on docs generation, logos, or tags. + +.. code-block:: yaml + + connection-types: + - hook-class-name: airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIAzureHook + hook-name: "Pydantic AI (Azure OpenAI)" + connection-type: pydanticai-azure + external-services: + - Azure OpenAI + conn-fields ~~~~~~~~~~~ diff --git a/dev/MANUALLY_BUILDING_IMAGES.md b/dev/MANUALLY_BUILDING_IMAGES.md index 18ba10dd210a8..aa0c8a4c14f77 100644 --- a/dev/MANUALLY_BUILDING_IMAGES.md +++ b/dev/MANUALLY_BUILDING_IMAGES.md @@ -94,7 +94,7 @@ The benefit of this method is that the images can be built and pushed in a singl to build and push them separately and perform the additional manifest merge step. ```bash -docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder (you might want to use +docker buildx create --name airflow_cache --driver docker-container unix:///var/run/docker.sock # your local builder docker buildx create --name airflow_cache --append HOST:PORT # your remote builder ``` @@ -167,7 +167,7 @@ with appropriate versions). You can then merge the images into a single multi-platform image with: ```bash -docker release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --airflow-version "${VERSION}" --metadata-folder dist ``` The same can be repeated for slim images by adding `--slim-images` option to the command, the manifests are @@ -189,5 +189,5 @@ breeze release-management release-prod-images --slim-images --airflow-version "$ Merging the images: ```bash -docker release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist +breeze release-management merge-prod-images --slim-images --airflow-version "${VERSION}" --metadata-folder dist ``` diff --git a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md index 01bdd7f75043d..cd07deec80bed 100644 --- a/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md +++ b/dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md @@ -25,8 +25,9 @@ - [Automated image cache and constraints refreshing in CI](#automated-image-cache-and-constraints-refreshing-in-ci) - [Manually refreshing the image cache](#manually-refreshing-the-image-cache) - [Why we need to update image cache manually](#why-we-need-to-update-image-cache-manually) - - [Prerequisites](#prerequisites) - - [How to refresh the image cache](#how-to-refresh-the-image-cache) + - [How to refresh the image cache via the CI workflow (recommended)](#how-to-refresh-the-image-cache-via-the-ci-workflow-recommended) + - [Prerequisites for refreshing locally](#prerequisites-for-refreshing-locally) + - [How to refresh the image cache locally](#how-to-refresh-the-image-cache-locally) - [Is it safe to refresh the image cache?](#is-it-safe-to-refresh-the-image-cache) - [What the command does](#what-the-command-does) - [Manually generating constraint files](#manually-generating-constraint-files) @@ -69,17 +70,17 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. * The latest [constraints](/contributing-docs/13_airflow_dependencies_and_extras.rst#pinned-constraint-files) are pushed to appropriate branch after all tests succeed in the `canary` build. -* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed early at the beginning of the - `canary` build. This is done twice during the canary build: - * By the `Push Early Image Cache` job that is run at the beginning of the `canary` build. This cover the - case when there are new dependencies added or Dockerfile/scripts change. Thanks to that step, subsequent - PRs will be faster when they use the new Dockerfile/script. Those jobs **might fail** occasionally, - if the latest PR added some conflicting dependencies with current constraints. This is not a problem - and when it happens, it will be fixed by the next step. - * By the `Push Image Cache` job that is run at the end of the `canary` build. This covers the case when - cache is also refreshed after than `main` build succeeds after the new constraints are pushed. This - step makes sure that constraints are committed and pushed just before the cache is refreshed, so - there is no problem with conflicting dependencies. +* The [images](breeze/doc/ci/02_images.md) in `ghcr.io` registry are refreshed by the `Push Image Cache` + job at the end of the `canary` build, once the tests have passed. Running it last is what makes the + constraints and the cache agree: the constraints are committed and pushed just before the cache is + refreshed from them. + +* Branches that are built on `push` rather than by the scheduled `canary` -- the release-prep + (`vX-Y-test`) and providers branches -- are refreshed by the + [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow + instead. It runs on its own rather than inside the CI run, because the CI run is cancelled by the + next push and a cache refresh that keeps being cancelled never lands. It covers every Python version + on both platforms, and it is the same workflow you run by hand (see below). # Manually refreshing the image cache @@ -87,14 +88,35 @@ rebuilding of [Breeze](./breeze/doc/README.rst) images for development purpose. ## Why we need to update image cache manually Sometimes, when we have a problem with our CI running and flakiness of GitHub Actions runners or our -tests, the refresh might not be triggered. This has been mitigated by "Push Early Image Cache" job added in -our CI, but there are other reasons you might want to refresh the cache. Sometimes we want to refresh the +tests, the refresh might not be triggered. Sometimes we want to refresh the image cache in `vX_Y_test` branch (following our convention of branch names `vX_Y_test` branch is the branch used to release all `X.Y.*` versions of airflow) before we attempt to push a change there. There are no PRs happening in this branch, so manual refresh before we make a PR might speed up the PR build. Or sometimes we just refreshed the constraints (see below) and we want the cache to include those. -## Prerequisites +## How to refresh the image cache via the CI workflow (recommended) + +Run the [`Refresh image registry cache`](../.github/workflows/refresh-image-registry-cache.yml) workflow +from the Actions tab, selecting the branch whose cache you want to refresh in GitHub's branch dropdown -- +the cache is namespaced per branch, so the branch you pick is the namespace written to. The `platform` +input refreshes `linux/amd64`, `linux/arm64`, or both (the default). + +Prefer this over the local route below. The workflow needs no buildx / qemu setup on your machine, it +does not depend on your upload bandwidth, it always covers every Python version, and it runs with the +registry credentials the CI already has, so you do not need to be logged in to `ghcr.io` as a committer. + +A run you start by hand is not cancelled by merges landing while it works -- it uses a concurrency +group of its own for that reason. Starting it again for the same branch does replace the run in +flight. Push-triggered refreshes are never cancelled either; a newer one waits for the one in +flight to finish. + +The workflow definition has to exist on the branch you select, so a release branch needs the workflow +backported before it can be refreshed there. + +## Prerequisites for refreshing locally + +The rest of this chapter describes refreshing the cache from your own machine. It is a fallback for when +the workflow cannot be used at all -- it is slower, and easy to get subtly wrong. Note that in order to refresh images you have to not only have `buildx` command installed for docker, but you should also make sure that you have the buildkit builder configured and set. @@ -146,7 +168,7 @@ docker buildx ls airflow_cache1 tcp://127.0.0.1:2375 ``` -## How to refresh the image cache +## How to refresh the image cache locally The images can be rebuilt and refreshed after the constraints are pushed. Refreshing image for all python version is as simple as running the [refresh_images.sh](refresh_images.sh) script which will @@ -217,11 +239,17 @@ and you need to be sure what you are doing, but you can always do it manually if ## How to refresh constraints via the CI workflow (recommended) The easiest way to refresh the constraints - for example to pick up newly released -providers/dependencies from PyPI just before promoting an RC - is to trigger the -[`Update constraints`](../.github/workflows/update-constraints-on-push.yml) workflow manually -instead of running the `breeze` commands locally. The workflow runs exactly the same steps -that run automatically when `uv.lock` changes, builds the CI images, generates all constraint -flavours and commits/pushes them to the matching `constraints-*` branch. +providers/dependencies from PyPI just before promoting an RC - is to run the +[`Refresh constraints`](../.github/workflows/refresh-constraints.yml) workflow instead of +running the `breeze` commands locally. It runs exactly the same steps that run automatically +when `uv.lock` changes (it calls that workflow), builds the CI images, and commits/pushes to the +matching `constraints-*` branch. + +It refreshes **all three constraint flavours**, not only the PyPI one: +`constraints-source-providers-X.Y.txt` (providers from the sources, which is what CI and Breeze +install), `constraints-no-providers-X.Y.txt` (core alone) and `constraints-X.Y.txt` (providers as +published on PyPI, which is what users install). Refreshing only the PyPI flavour would leave CI +and Breeze pinned to the older versions, so the three would disagree about the same dependency. The manual run is always launched from `main` and takes a `ref` input that selects the commit-ish (branch, tag or commit hash) whose sources the constraints are refreshed from. You @@ -236,13 +264,13 @@ workflow); the automatic `uv.lock`-push runs are not restricted. To run it: -![Run the Update constraints workflow](images/update_constraints_run_workflow.png) +![Run the Refresh constraints workflow](images/update_constraints_run_workflow.png) -1. Go to the [`Update constraints`](https://github.com/apache/airflow/actions/workflows/update-constraints-on-push.yml) +1. Go to the [`Refresh constraints`](https://github.com/apache/airflow/actions/workflows/refresh-constraints.yml) workflow in the Actions tab. 2. Click **Run workflow** and keep the branch set to `main` (this is where the workflow runs from - it is not the branch whose constraints get refreshed). -3. In the **Optional repo reference to build constraints** field, enter the ref to refresh +3. In the **Repo reference to refresh constraints from** field, enter the ref to refresh constraints from - for example `v3-3-test`, `v3-3-stable`, `constraints-3-3`, an RC tag, or a commit hash. Leave it empty to refresh the branch you run from (`main` -> `constraints-main`). The matching `constraints-X-Y` branch to push to is derived automatically from that ref's diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md index 33881515df933..6f140f3126ebc 100644 --- a/dev/README_RELEASE_AIRFLOW.md +++ b/dev/README_RELEASE_AIRFLOW.md @@ -588,6 +588,18 @@ still works but is no longer recommended. --sync-branch ${SYNC_BRANCH} ``` + Note: when it reaches the constraints step, `start-rc-process` triggers the `Release + constraints` workflow and waits. The candidate resolves constraints of its own rather than + tagging the `constraints-X-Y` branch tip, and it resolves them **allowing pre-releases for the + providers** — the providers of the wave being voted on exist on PyPI only as `rcN` versions, so + constraints that refused them could not describe what a tester is asked to install. The + allowance is scoped to `apache-airflow-providers-*`; no other package can resolve to a + pre-release, so a beta of some third-party library cannot slip into what the candidate ships. + + The result lands on a branch of its own (`constraints-${VERSION_RC}`) and is tagged + `constraints-${VERSION_RC}`. The shared `constraints-X-Y` branch is left where it was — only + the final release moves it. + **Testing the start-rc-process command:** Before running the actual release command, you can safely test it using: @@ -1372,15 +1384,44 @@ breeze release-management start-release \ Note: The `--task-sdk-version` parameter is optional. If you are releasing Airflow without a corresponding Task SDK release, you can omit this parameter. -Note: When it reaches the constraints step, `start-release` asks whether to base the final -`constraints-${VERSION}` tag on the latest `constraints-X-Y` branch tip instead of the RC -constraints tag. The RC constraints are frozen when the RC is cut, so if any providers were -released (or constraints were otherwise refreshed - see +Note: When it reaches the constraints step, `start-release` resolves the constraints again rather +than promoting the ones the RC was cut with. The RC constraints deliberately pin pre-releases - +that wave's providers exist on PyPI only as `rcN` versions at candidate time - so they can never +become the released constraints by retagging. The regeneration runs without pre-releases against +the same providers now published as finals, commits the result onto the `constraints-X-Y` branch, +pushes it, and tags it `constraints-${VERSION}`. That commit is what makes the released +constraints the new baseline, so refreshing the branch beforehand (see [MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md](MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md)) -after the last RC and you want the released constraints to reflect that, answer **yes** to tag the -`constraints-X-Y` branch tip. Otherwise (the default) the final tag matches the RC exactly. If you -do refresh, run the `Update constraints` workflow from `main` with `ref` set to the ref you are -releasing (typically `v3-*-stable`) **before** running `start-release`. +is no longer necessary. + +The resolution runs on CI runners through the `Release constraints` workflow, so the release +manager's machine does not need a CI image for every supported Python. `start-release` triggers it +and waits. You can also run it on its own - to redo a candidate's constraints, or to produce them +for a release cut before this existed: + +```shell script +breeze workflow-run release-constraints --version ${VERSION} --ref v3-1-stable +``` + +The workflow derives the stage from `--version` alone, so there is no separate switch that could +disagree with the version: + +| `--version` | Provider pins | Lands on | Tagged | +|---|---|---|---| +| `3.3.1rc1` | newest in PyPI, `rcN` included | `constraints-3.3.1rc1`, branched off `constraints-3-3` | `constraints-3.3.1rc1` | +| `3.3.1` | newest final in PyPI | `constraints-3-3` (commit) | `constraints-3.3.1` | + +The providers are pinned rather than resolved: every `apache-airflow-providers-*` is named with +`==` at the newest version PyPI can install, so the constraints record what is actually published +instead of whatever the resolver settles on. A candidate takes the wave's `rcN` versions along with +it - a pre-release only wins by sorting above every final release, so a provider without a +candidate in the wave keeps its release. A final ignores candidates entirely, which is what makes +it impossible for a released constraints file to carry an `rc` pin. Nothing else in the dependency +graph can resolve to a pre-release, since no other requirement mentions one. + +Re-running the workflow for the same candidate deletes and re-creates that candidate's branch and +tag, so redoing a candidate's constraints just works. A final is never treated this way - it commits +onto the shared `constraints-X-Y` branch, whose history everything downstream reads. 4. Make sure to update Airflow version in ``v3-*-test`` branch after cherry-picking to X.Y.1 in diff --git a/dev/README_RELEASE_PYTHON_CLIENT.md b/dev/README_RELEASE_PYTHON_CLIENT.md index 8d29d7e043258..88e2a8b4d14e6 100644 --- a/dev/README_RELEASE_PYTHON_CLIENT.md +++ b/dev/README_RELEASE_PYTHON_CLIENT.md @@ -760,8 +760,8 @@ Give the server 20-30 seconds to serialize the example Dags to DB 3. In the meantime install the python client you want to test - in the terminal window in the container. It can be installed from `PyPI` via `pip install apache-airflow-client==X.Y.Zrc1` or installed - from a file (for example if you nust built the client with `breeze release-management prepare-python-client`, - there will be `/dist/apache_airflow_client-py*.whl` file in in your `/dist` folder (mapped from `dist` + from a file (for example if you just built the client with `breeze release-management prepare-python-client`, + there will be `/dist/apache_airflow_client-py*.whl` file in your `/dist` folder (mapped from `dist` folder in your checked out airflow repository and you can install it with: `pip install /dist/apache_airflow_client-py*.whl`. diff --git a/dev/breeze/doc/09_release_management_tasks.rst b/dev/breeze/doc/09_release_management_tasks.rst index 70453de37db6e..40694e437c660 100644 --- a/dev/breeze/doc/09_release_management_tasks.rst +++ b/dev/breeze/doc/09_release_management_tasks.rst @@ -1080,6 +1080,27 @@ These are all available flags of ``workflow-run publish-docs`` command: :width: 100% :alt: Breeze workflow-run publish-docs +Resolving the constraints for a release +""""""""""""""""""""""""""""""""""""""" + +To trigger the GitHub Actions workflow that resolves, publishes and tags the constraints belonging to a +release, you can use the ``breeze workflow-run release-constraints`` command. The release commands trigger +it themselves, so this is for redoing a candidate's constraints or producing them for a release cut before +the workflow existed. + +The stage is derived from ``--version`` alone, so it cannot be set inconsistently with it. A candidate +(``3.1.3rc1``) resolves allowing pre-releases for ``apache-airflow-providers-*`` — the providers of the wave +being voted on are on PyPI only as ``rcN`` versions — and lands on a branch of its own, leaving the shared +``constraints-X-Y`` branch where it was. A final (``3.1.3``) resolves without them and commits onto +``constraints-X-Y``, which is what makes the released constraints the baseline everything downstream reads. + +These are all available flags of ``workflow-run release-constraints`` command: + +.. image:: ./images/output_workflow-run_release-constraints.svg + :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_workflow-run_release-constraints.svg + :width: 100% + :alt: Breeze workflow-run release-constraints + Publishing the schema files to S3 """"""""""""""""""""""""""""""""" diff --git a/dev/breeze/doc/images/output_release-management_generate-constraints.svg b/dev/breeze/doc/images/output_release-management_generate-constraints.svg index 9a2a1328ea1d0..07e3538c1cef6 100644 --- a/dev/breeze/doc/images/output_release-management_generate-constraints.svg +++ b/dev/breeze/doc/images/output_release-management_generate-constraints.svg @@ -1,4 +1,4 @@ - + diff --git a/dev/breeze/doc/images/output_release-management_generate-constraints.txt b/dev/breeze/doc/images/output_release-management_generate-constraints.txt index 6b4ef06da4a35..6328985d0eebb 100644 --- a/dev/breeze/doc/images/output_release-management_generate-constraints.txt +++ b/dev/breeze/doc/images/output_release-management_generate-constraints.txt @@ -1 +1 @@ -8f5434ddc6564b1a2b026a63424eecea +ab2a920a9c7ec445e7edd70051af73e1 diff --git a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg index 202eba652be36..938ff18d35521 100644 --- a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg +++ b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.svg @@ -1,4 +1,4 @@ - + diff --git a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt index 92e75f17fd456..6987a59efd88f 100644 --- a/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt +++ b/dev/breeze/doc/images/output_setup_check-all-params-in-groups.txt @@ -1 +1 @@ -fc5f65d76377357873c6b3ea200f2ec8 +b52151d5b1fb9ecfd92edf910750f0ee diff --git a/dev/breeze/doc/images/output_setup_regenerate-command-images.svg b/dev/breeze/doc/images/output_setup_regenerate-command-images.svg index de56522e8180b..8b96e56ba30a2 100644 --- a/dev/breeze/doc/images/output_setup_regenerate-command-images.svg +++ b/dev/breeze/doc/images/output_setup_regenerate-command-images.svg @@ -261,7 +261,7 @@ │testing:helm-tests | testing:providers-integration-tests | testing:providers-tests | │ │testing:python-api-client-tests | testing:system-tests | testing:task-sdk-integration-tests | │ │testing:task-sdk-tests | testing:ui-e2e-tests | ui | ui:check-translation-completeness | │ -│ui:compile-assets | workflow-run | workflow-run:publish-docs)│ +│ui:compile-assets | workflow-run | workflow-run:publish-docs | workflow-run:release-constraints)│ │--check-onlyOnly check if some images need to be regenerated. Return 0 if no need or 1 if needed. Cannot be used  │ │together with --command flag or --force.                                                              │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/dev/breeze/doc/images/output_setup_regenerate-command-images.txt b/dev/breeze/doc/images/output_setup_regenerate-command-images.txt index fe2e502097d32..ac7e52d52bf16 100644 --- a/dev/breeze/doc/images/output_setup_regenerate-command-images.txt +++ b/dev/breeze/doc/images/output_setup_regenerate-command-images.txt @@ -1 +1 @@ -141b4bc2fbf13eddd1301de1064bf9fe +fadc598c6c80132f76537b23c462e6ed diff --git a/dev/breeze/doc/images/output_workflow-run.svg b/dev/breeze/doc/images/output_workflow-run.svg index af211f19a8e77..f8f7cb779f000 100644 --- a/dev/breeze/doc/images/output_workflow-run.svg +++ b/dev/breeze/doc/images/output_workflow-run.svg @@ -1,4 +1,4 @@ - + │--help-hShow this message and exit.│ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─ Airflow github actions workflow commands ───────────────────────────────────────────────────────────────────────────╮ -│publish-docs                  Trigger publish docs to S3 workflow                                                 │ -╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +│publish-docs         Trigger publish docs to S3 workflow                                                          │ +│release-constraints  Trigger the workflow that resolves, publishes and tags the constraints for a release.        │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ diff --git a/dev/breeze/doc/images/output_workflow-run.txt b/dev/breeze/doc/images/output_workflow-run.txt index 096f1b4e2260f..606ed71654d69 100644 --- a/dev/breeze/doc/images/output_workflow-run.txt +++ b/dev/breeze/doc/images/output_workflow-run.txt @@ -1 +1 @@ -348e7af5d380b4c7dd9452e48dfc4abf +cfbf0a5234b3a9689a01d164e3e91abf diff --git a/dev/breeze/doc/images/output_workflow-run_release-constraints.svg b/dev/breeze/doc/images/output_workflow-run_release-constraints.svg new file mode 100644 index 0000000000000..e20e2d2696b27 --- /dev/null +++ b/dev/breeze/doc/images/output_workflow-run_release-constraints.svg @@ -0,0 +1,138 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Command: workflow-run release-constraints + + + + + + + + + + +Usage:breeze workflow-run release-constraints[OPTIONS] + +Trigger the workflow that resolves, publishes and tags the constraints for a release. + +╭─ Constraints to resolve ─────────────────────────────────────────────────────────────────────────────────────────────╮ +│*--version        Version the constraints belong to. A candidate (3.1.3rc1) resolves with pre-releases allowed  │ +│and lands on a branch of its own; a final (3.1.3) resolves without them and commits onto      │ +│constraints-X-Y. The stage is derived from this, so it cannot be set inconsistently.          │ +│[required](TEXT)│ +│*--ref            Git ref the constraints are resolved from, e.g. 'v3-1-stable' or the release tag. [required]│ +│(TEXT)│ +│--workflow-branchGit ref the workflow DEFINITION runs from. Defaults to 'main', which is normally what you     │ +│want: unlike the docs build, the constraints do not have to be produced by the workflow as it │ +│stood at the ref being released. [default: main](TEXT)│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Common options ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│--help-hShow this message and exit.│ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ + + + + diff --git a/dev/breeze/doc/images/output_workflow-run_release-constraints.txt b/dev/breeze/doc/images/output_workflow-run_release-constraints.txt new file mode 100644 index 0000000000000..871ae3667b4a6 --- /dev/null +++ b/dev/breeze/doc/images/output_workflow-run_release-constraints.txt @@ -0,0 +1 @@ +14273c8c9c667e0004c0a1379d53da58 diff --git a/dev/breeze/pyproject.toml b/dev/breeze/pyproject.toml index ccbd0e107272b..4ab83e7901055 100644 --- a/dev/breeze/pyproject.toml +++ b/dev/breeze/pyproject.toml @@ -47,8 +47,12 @@ dependencies = [ "black>=26.1.0", "click>=8.3.0", "filelock>=3.13.0", - "flit>=3.12.0", - "flit-core>=3.12.0", + # Capped below 4 because `flit build` has no PEP 517 isolation: the installed flit_core is + # what builds every provider distribution, so the `flit_core==` pin they declare never + # applies. Adopting the 4.x backend is tracked at + # https://github.com/apache/airflow/issues/71121 + "flit>=3.12.0,<4", + "flit-core>=3.12.0,<4", "google-api-python-client>=2.142.0", "google-auth-httplib2>=0.2.0", "google-auth-oauthlib>=1.2.0", diff --git a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py index 5175e6e40ce76..f6ac278bb35ae 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py +++ b/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py @@ -49,6 +49,7 @@ AIRFLOW_ROOT_PATH, OUT_PATH, ) +from airflow_breeze.utils.release_constraints import publish_constraints from airflow_breeze.utils.reproducible import get_source_date_epoch, repack_deterministically from airflow_breeze.utils.run_utils import run_command from airflow_breeze.utils.shared_options import get_dry_run @@ -442,28 +443,10 @@ def sign_the_release(repo_root): console_print("[success]Release signed") -def tag_and_push_constraints(version, version_branch, remote_name): - if confirm_action("Do you want to tag and push constraints?"): - run_command( - ["git", "checkout", f"{remote_name}/constraints-{version_branch}"], - check=True, - ) - run_command( - [ - "git", - "tag", - "-s", - f"constraints-{version}", - "-m", - f"Constraints for Apache Airflow {version}", - ], - check=True, - ) - run_command( - ["git", "push", remote_name, "tag", f"constraints-{version}"], - check=True, - ) - console_print("[success]Constraints tagged and pushed") +def generate_and_push_constraints(version, version_branch): + # Resolved from the stable branch the candidate was cut from, so the constraints describe the + # sources being voted on. The workflow reads "rcN" and allows pre-releases accordingly. + publish_constraints(version=version, ref=f"v{version_branch}-stable") def clone_asf_repo(version, repo_root): @@ -845,8 +828,8 @@ def publish_release_candidate( test_airflow() # Sign the release sign_the_release(airflow_repo_root) - # Tag and push constraints - tag_and_push_constraints(version, version_branch, remote_name) + # Generate, publish and tag the constraints for this candidate + generate_and_push_constraints(version, version_branch) # Clone the asf repo clone_asf_repo(version, airflow_repo_root) # Move artifacts to SVN diff --git a/dev/breeze/src/airflow_breeze/commands/release_command.py b/dev/breeze/src/airflow_breeze/commands/release_command.py index 1b93f9c28ac43..ffa97a01f6326 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_command.py +++ b/dev/breeze/src/airflow_breeze/commands/release_command.py @@ -29,6 +29,7 @@ from airflow_breeze.utils.console import console_print from airflow_breeze.utils.environment_check import is_ci_environment from airflow_breeze.utils.path_utils import AIRFLOW_ROOT_PATH +from airflow_breeze.utils.release_constraints import publish_constraints from airflow_breeze.utils.run_utils import run_command # Pattern to match Airflow release versions (e.g., "3.0.5") @@ -343,45 +344,12 @@ def upload_to_pypi(version, task_sdk_version=None): ) -def get_constraints_branch_for_version(version: str) -> str: +def regenerate_constraints(version): + # The candidate's constraints pinned pre-releases, so they cannot be promoted by retagging - a + # released version must never point at rc pins. The workflow reads a version without "rcN" and + # resolves again without them, against the providers now published as finals. major, minor = version.split(".")[:2] - return f"constraints-{major}-{minor}" - - -def retag_constraints(release_candidate, version): - # By default the final ``constraints-`` tag is created from the RC constraints tag. - # If the constraints were refreshed after the last RC (e.g. to pick up newly released - # providers - see dev/MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md) the tip of the - # ``constraints-X-Y`` branch is newer than the RC tag and should be tagged instead. - constraints_branch = get_constraints_branch_for_version(version) - source_ref = f"constraints-{release_candidate}" - if confirm_action( - f"Base the final constraints on the latest '{constraints_branch}' branch tip instead of the " - f"'{source_ref}' tag? Choose yes if you refreshed constraints after {release_candidate}." - ): - run_command(["git", "fetch", "origin", constraints_branch], check=True) - source_ref = f"origin/{constraints_branch}" - if confirm_action(f"Retag constraints from {source_ref} as {version}?"): - run_command( - ["git", "checkout", source_ref], - check=True, - ) - run_command( - [ - "git", - "tag", - "-s", - f"constraints-{version}", - "-m", - f"Constraints for Apache Airflow {version}", - ], - check=True, - ) - if confirm_action(f"Push constraints-{version} tag to GitHub?"): - run_command( - ["git", "push", "origin", "tag", f"constraints-{version}"], - check=True, - ) + publish_constraints(version=version, ref=f"v{major}-{minor}-stable") def tag_and_push_latest_constraint(version): @@ -567,8 +535,8 @@ def airflow_release(version, task_sdk_version): # Change Directory to airflow os.chdir(airflow_repo_root) - # Retag and push the constraint file - retag_constraints(release_candidate, version) + # Regenerate, publish and tag the constraints for the final version + regenerate_constraints(version) tag_and_push_latest_constraint(version) # Push tag for final version diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py index c7efa5a2fd3d6..52ed5a6001ac1 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py @@ -1548,6 +1548,7 @@ def tag_providers( @option_debug_resources @option_python_versions @option_airflow_constraints_mode_ci +@option_allow_pre_releases @option_github_repository @option_use_uv @option_verbose @@ -1555,6 +1556,7 @@ def tag_providers( @option_answer def generate_constraints( airflow_constraints_mode: str, + allow_pre_releases: bool, debug_resources: bool, github_repository: str, parallelism: int, @@ -1599,6 +1601,7 @@ def generate_constraints( shell_params_list = [ ShellParams( airflow_constraints_mode=airflow_constraints_mode, + allow_pre_releases=allow_pre_releases, github_repository=github_repository, python=python, use_uv=use_uv, @@ -1617,6 +1620,7 @@ def generate_constraints( else: shell_params = ShellParams( airflow_constraints_mode=airflow_constraints_mode, + allow_pre_releases=allow_pre_releases, github_repository=github_repository, python=python, use_uv=use_uv, diff --git a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py index 2acad7f44f859..ac59513cdfb91 100644 --- a/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands_config.py @@ -339,6 +339,7 @@ "name": "Generate constraints flags", "options": [ "--airflow-constraints-mode", + "--allow-pre-releases", "--github-repository", "--python", "--use-uv", diff --git a/dev/breeze/src/airflow_breeze/commands/workflow_commands.py b/dev/breeze/src/airflow_breeze/commands/workflow_commands.py index 2330edfcba6ab..d6c98827b4252 100644 --- a/dev/breeze/src/airflow_breeze/commands/workflow_commands.py +++ b/dev/breeze/src/airflow_breeze/commands/workflow_commands.py @@ -34,8 +34,13 @@ "publish-docs": "publish-docs-to-s3.yml", "airflow-refresh-site": "build.yml", "sync-s3-to-github": "s3-to-github.yml", + "release-constraints": "release-constraints.yml", } +# X.Y.Z or X.Y.ZrcN - the workflow derives the release stage from which of the two it is given, +# so there is no separate switch that could disagree with the version. +RELEASE_VERSION_PATTERN = re.compile(r"^\d+\.\d+\.\d+(rc\d+)?$") + APACHE_AIRFLOW_REPO = "apache/airflow" APACHE_AIRFLOW_SITE_REPO = "apache/airflow-site" APACHE_AIRFLOW_SITE_ARCHIVE_REPO = "apache/airflow-site-archive" @@ -272,3 +277,42 @@ def workflow_run_publish( **workflow_fields, monitor=False, ) + + +@workflow_run_group.command( + name="release-constraints", + help="Trigger the workflow that resolves, publishes and tags the constraints for a release.", +) +@click.option( + "--version", + help="Version the constraints belong to. A candidate (3.1.3rc1) resolves with pre-releases " + "allowed and lands on a branch of its own; a final (3.1.3) resolves without them and commits " + "onto constraints-X-Y. The stage is derived from this, so it cannot be set inconsistently.", + required=True, +) +@click.option( + "--ref", + help="Git ref the constraints are resolved from, e.g. 'v3-1-stable' or the release tag.", + required=True, +) +@click.option( + "--workflow-branch", + help="Git ref the workflow DEFINITION runs from. Defaults to 'main', which is normally what " + "you want: unlike the docs build, the constraints do not have to be produced by the workflow " + "as it stood at the ref being released.", + default="main", + show_default=True, +) +def workflow_run_release_constraints(version: str, ref: str, workflow_branch: str): + if not RELEASE_VERSION_PATTERN.match(version): + console_print(f"[red]Error: '{version}' is not a release version - expected X.Y.Z or X.Y.ZrcN.[/red]") + sys.exit(1) + stage = "candidate" if "rc" in version else "final" + console_print(f"[blue]Triggering constraints generation for the {stage} {version} from {ref}[/blue]") + trigger_workflow_and_monitor( + workflow_name=WORKFLOW_NAME_MAPS["release-constraints"], + repo=APACHE_AIRFLOW_REPO, + branch=workflow_branch, + version=version, + ref=ref, + ) diff --git a/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py b/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py index 239a5e46a6a84..9c6d0c9927764 100644 --- a/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py +++ b/dev/breeze/src/airflow_breeze/commands/workflow_commands_config.py @@ -18,7 +18,7 @@ WORKFLOW_RUN_COMMANDS: dict[str, str | list[str]] = { "name": "Airflow github actions workflow commands", - "commands": ["publish-docs"], + "commands": ["publish-docs", "release-constraints"], } WORKFLOW_RUN_PARAMETERS: dict[str, list[dict[str, str | list[str]]]] = { @@ -55,3 +55,13 @@ }, ], } +WORKFLOW_RUN_PARAMETERS["breeze workflow-run release-constraints"] = [ + { + "name": "Constraints to resolve", + "options": [ + "--version", + "--ref", + "--workflow-branch", + ], + }, +] diff --git a/dev/breeze/src/airflow_breeze/global_constants.py b/dev/breeze/src/airflow_breeze/global_constants.py index 8f90bb9ca19a1..8db570e63da5e 100644 --- a/dev/breeze/src/airflow_breeze/global_constants.py +++ b/dev/breeze/src/airflow_breeze/global_constants.py @@ -502,6 +502,7 @@ def get_default_platform_machine() -> str: REDIS_HOST_PORT = "26379" RABBITMQ_HOST_PORT = "25672" SSH_PORT = "12322" +SIMPLE_AUTH_MANAGER_VITE_DEV_PORT = "5174" VITE_DEV_PORT = "5173" WEB_HOST_PORT = "28080" BREEZE_DEBUG_SCHEDULER_PORT = "50231" diff --git a/dev/breeze/src/airflow_breeze/prepare_providers/provider_distributions.py b/dev/breeze/src/airflow_breeze/prepare_providers/provider_distributions.py index ff4ce0b0a0332..459b66a013ccf 100644 --- a/dev/breeze/src/airflow_breeze/prepare_providers/provider_distributions.py +++ b/dev/breeze/src/airflow_breeze/prepare_providers/provider_distributions.py @@ -251,7 +251,7 @@ def build_provider_distribution( build_backend = provider_info.get("build-system", "flit_core") build_env = {"SOURCE_DATE_EPOCH": str(get_provider_details(provider_id).source_date_epoch)} if build_backend == "flit_core": - command: list[str] = [sys.executable, "-m", "flit", "build", "--no-setup-py", "--use-vcs"] + command: list[str] = [sys.executable, "-m", "flit", "build", "--use-vcs"] console_print( "[warning]Workaround wheel-only package bug in flit by building both and removing sdist." ) diff --git a/dev/breeze/src/airflow_breeze/utils/gh_workflow_utils.py b/dev/breeze/src/airflow_breeze/utils/gh_workflow_utils.py index caaa7b5ac7859..78d19080ea5d1 100644 --- a/dev/breeze/src/airflow_breeze/utils/gh_workflow_utils.py +++ b/dev/breeze/src/airflow_breeze/utils/gh_workflow_utils.py @@ -26,6 +26,7 @@ from airflow_breeze.global_constants import MIN_GH_VERSION from airflow_breeze.utils.console import console_print from airflow_breeze.utils.github import run_gh_command +from airflow_breeze.utils.shared_options import get_dry_run def tigger_workflow(workflow_name: str, repo: str, branch: str = "main", **kwargs): @@ -56,6 +57,11 @@ def tigger_workflow(workflow_name: str, repo: str, branch: str = "main", **kwarg console_print(f"[red]Error running workflow: {result.stderr}[/red]") sys.exit(1) + if get_dry_run(): + # A dry run dispatches nothing, so `gh run list` comes back empty. + console_print(f"[info]Dry run: not looking up or monitoring a run of {workflow_name}.") + return + # Wait for a few seconds to start the workflow run time.sleep(5) @@ -204,6 +210,9 @@ def trigger_workflow_and_monitor( **workflow_fields, ) + if get_dry_run(): + return + workflow_run_id = get_workflow_run_id( workflow_name=workflow_name, repo=repo, diff --git a/dev/breeze/src/airflow_breeze/utils/release_constraints.py b/dev/breeze/src/airflow_breeze/utils/release_constraints.py new file mode 100644 index 0000000000000..103f45e750b91 --- /dev/null +++ b/dev/breeze/src/airflow_breeze/utils/release_constraints.py @@ -0,0 +1,62 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Handing the constraints of a release over to the workflow that resolves them. + +A release used to tag whatever the ``constraints-X-Y`` branch happened to hold, so the constraints +shipped with a version were a resolution made by the last CI build rather than one made for that +version. They are resolved per release now, by the ``release-constraints`` workflow - on runners +rather than on the release manager's machine, which would otherwise need a CI image for every +supported Python before it could cut a release. + +The workflow derives the stage from the version it is given, so nothing here has to pass a +separate switch that could disagree with it. +""" + +from __future__ import annotations + +from airflow_breeze.utils.confirm import confirm_action +from airflow_breeze.utils.console import console_print +from airflow_breeze.utils.gh_workflow_utils import trigger_workflow_and_monitor + +RELEASE_CONSTRAINTS_WORKFLOW = "release-constraints.yml" +APACHE_AIRFLOW_REPO = "apache/airflow" + + +def publish_constraints(*, version: str, ref: str, workflow_branch: str = "main") -> None: + """Resolve, publish and tag the constraints belonging to ``version``. + + ``version`` alone decides what happens: a candidate (``3.3.1rc1``) pins the providers at the + versions PyPI holds - the wave being voted on is there only as rc versions - and lands on a + branch of its own, leaving the branch every other build reads where it was. Re-running it for + the same candidate replaces that branch and its tag, so a wave can be re-cut. A final + (``3.3.1``) resolves against the published releases and commits onto ``constraints-X-Y``, which + is what makes the released constraints the baseline everything downstream reads. + """ + stage = "candidate" if "rc" in version else "final" + if not confirm_action( + f"Trigger the release-constraints workflow for the {stage} {version} (resolved from {ref})?" + ): + return + console_print(f"[info]Resolving constraints for {version} from {ref} - this runs on CI runners.") + trigger_workflow_and_monitor( + workflow_name=RELEASE_CONSTRAINTS_WORKFLOW, + repo=APACHE_AIRFLOW_REPO, + branch=workflow_branch, + version=version, + ref=ref, + ) + console_print(f"[success]Constraints for {version} published and tagged 'constraints-{version}'") diff --git a/dev/breeze/src/airflow_breeze/utils/run_utils.py b/dev/breeze/src/airflow_breeze/utils/run_utils.py index d8c0550019a7b..c503841e97eb5 100644 --- a/dev/breeze/src/airflow_breeze/utils/run_utils.py +++ b/dev/breeze/src/airflow_breeze/utils/run_utils.py @@ -25,16 +25,18 @@ import shlex import shutil import signal +import socket import stat import subprocess import sys -from collections.abc import Mapping +from collections.abc import Iterable, Mapping from pathlib import Path from subprocess import CalledProcessError, CompletedProcess from typing import Any from rich.markup import escape +from airflow_breeze.global_constants import SIMPLE_AUTH_MANAGER_VITE_DEV_PORT, VITE_DEV_PORT from airflow_breeze.utils.ci_group import ci_group from airflow_breeze.utils.console import Output, console_print, get_console from airflow_breeze.utils.functools_cache import clearable_cache @@ -532,12 +534,30 @@ def _clean_ui_assets(additional_ui_hooks: list[str]): console_print("[success]Cleaned ui assets[/]") +def _find_occupied_local_ports(ports: Iterable[str]) -> list[str]: + occupied_ports = [] + for port in ports: + with contextlib.suppress(OSError): + with socket.create_connection(("localhost", int(port)), timeout=0.1): + occupied_ports.append(port) + return occupied_ports + + def run_compile_ui_assets( dev: bool, run_in_background: bool, force_clean: bool, additional_ui_hooks: list[str], ): + if dev: + occupied_ports = _find_occupied_local_ports((VITE_DEV_PORT, SIMPLE_AUTH_MANAGER_VITE_DEV_PORT)) + if occupied_ports: + console_print( + "[error]Cannot start UI development servers because the following local port(s) " + f"are already in use: {', '.join(occupied_ports)}.[/]\n" + "[info]Stop the processes using these ports and try again.[/]" + ) + sys.exit(1) if force_clean: _clean_ui_assets(additional_ui_hooks) if dev: diff --git a/dev/breeze/tests/test_gh_workflow_utils.py b/dev/breeze/tests/test_gh_workflow_utils.py new file mode 100644 index 0000000000000..084944a525f28 --- /dev/null +++ b/dev/breeze/tests/test_gh_workflow_utils.py @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +from unittest import mock + +from airflow_breeze.utils.gh_workflow_utils import trigger_workflow_and_monitor +from airflow_breeze.utils.shared_options import set_dry_run + + +@mock.patch("airflow_breeze.utils.gh_workflow_utils.monitor_workflow_run") +@mock.patch("airflow_breeze.utils.gh_workflow_utils.make_sure_gh_is_installed") +def test_trigger_workflow_and_monitor_stops_after_the_dispatch_in_dry_run(_, mock_monitor): + """A dry run dispatches nothing, so the empty `gh run list` must not read as a missing run.""" + set_dry_run(True) + try: + trigger_workflow_and_monitor( + workflow_name="release-constraints.yml", + repo="apache/airflow", + version="3.2.0rc1", + ref="v3-2-stable", + ) + finally: + set_dry_run(False) + + mock_monitor.assert_not_called() diff --git a/dev/breeze/tests/test_release_command.py b/dev/breeze/tests/test_release_command.py index 1e023e6bf5bfd..c1d89193e4ff5 100644 --- a/dev/breeze/tests/test_release_command.py +++ b/dev/breeze/tests/test_release_command.py @@ -540,53 +540,22 @@ def fake_getcwd() -> str: assert chdir_calls == [svn_release_repo, "/original/dir"] -@pytest.mark.parametrize( - ("version", "expected_branch"), - [ - ("3.3.0", "constraints-3-3"), - ("3.0.5", "constraints-3-0"), - ("3.12.10", "constraints-3-12"), - ], -) -def test_get_constraints_branch_for_version(release_cmd, version, expected_branch): - assert release_cmd.get_constraints_branch_for_version(version) == expected_branch - - -def test_retag_constraints_from_rc_tag_by_default(monkeypatch, release_cmd): - run_command_calls: list[list[str]] = [] - - def fake_confirm_action(prompt: str, **_kwargs) -> bool: - # Decline basing on the branch tip; accept the retag and the push. - return not prompt.startswith("Base the final constraints on the latest") - - monkeypatch.setattr(release_cmd, "confirm_action", fake_confirm_action) - monkeypatch.setattr(release_cmd, "run_command", lambda cmd, **_kwargs: run_command_calls.append(cmd)) - - release_cmd.retag_constraints("3.3.0rc2", "3.3.0") +def test_regenerate_constraints_resolves_from_the_stable_branch(monkeypatch, release_cmd): + """The final release resolves from the branch it is cut from, not from main.""" + calls: dict[str, object] = {} + monkeypatch.setattr(release_cmd, "publish_constraints", lambda **kw: calls.update(kw)) - assert ["git", "checkout", "constraints-3.3.0rc2"] in run_command_calls - assert ["git", "tag", "-s", "constraints-3.3.0", "-m", "Constraints for Apache Airflow 3.3.0"] in ( - run_command_calls - ) - assert ["git", "push", "origin", "tag", "constraints-3.3.0"] in run_command_calls - # The branch tip is not fetched when basing on the RC tag. - assert not any(cmd[:2] == ["git", "fetch"] for cmd in run_command_calls) + release_cmd.regenerate_constraints("3.3.0") + assert calls == {"version": "3.3.0", "ref": "v3-3-stable"} -def test_retag_constraints_from_branch_tip_when_confirmed(monkeypatch, release_cmd): - run_command_calls: list[list[str]] = [] - def fake_confirm_action(prompt: str, **_kwargs) -> bool: - return True +def test_regenerate_constraints_passes_a_final_version(monkeypatch, release_cmd): + """No rc suffix reaches the workflow, which is what makes it refuse pre-releases.""" + calls: dict[str, object] = {} + monkeypatch.setattr(release_cmd, "publish_constraints", lambda **kw: calls.update(kw)) - monkeypatch.setattr(release_cmd, "confirm_action", fake_confirm_action) - monkeypatch.setattr(release_cmd, "run_command", lambda cmd, **_kwargs: run_command_calls.append(cmd)) - - release_cmd.retag_constraints("3.3.0rc2", "3.3.0") + release_cmd.regenerate_constraints("3.0.5") - assert ["git", "fetch", "origin", "constraints-3-3"] in run_command_calls - assert ["git", "checkout", "origin/constraints-3-3"] in run_command_calls - assert ["git", "checkout", "constraints-3.3.0rc2"] not in run_command_calls - assert ["git", "tag", "-s", "constraints-3.3.0", "-m", "Constraints for Apache Airflow 3.3.0"] in ( - run_command_calls - ) + assert "rc" not in str(calls["version"]) + assert calls["ref"] == "v3-0-stable" diff --git a/dev/breeze/tests/test_release_constraints.py b/dev/breeze/tests/test_release_constraints.py new file mode 100644 index 0000000000000..db914d9772244 --- /dev/null +++ b/dev/breeze/tests/test_release_constraints.py @@ -0,0 +1,91 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import pytest + +from airflow_breeze.utils import release_constraints + + +@pytest.fixture +def triggered(monkeypatch): + """Capture what would have been dispatched, with the prompt answered yes.""" + calls: list[dict] = [] + monkeypatch.setattr(release_constraints, "confirm_action", lambda *a, **kw: True) + monkeypatch.setattr( + release_constraints, "trigger_workflow_and_monitor", lambda **kwargs: calls.append(kwargs) + ) + return calls + + +@pytest.mark.parametrize( + ("version", "ref"), + [ + pytest.param("3.1.3rc1", "v3-1-stable", id="candidate"), + pytest.param("3.1.3", "v3-1-stable", id="final"), + ], +) +def test_version_and_ref_are_passed_through(triggered, version, ref): + release_constraints.publish_constraints(version=version, ref=ref) + + assert len(triggered) == 1 + assert triggered[0]["version"] == version + assert triggered[0]["ref"] == ref + assert triggered[0]["workflow_name"] == "release-constraints.yml" + assert triggered[0]["repo"] == "apache/airflow" + + +def test_workflow_definition_defaults_to_main(triggered): + # Unlike the docs build, constraints need not be produced by the workflow as it stood at the + # ref being released - so this defaults to main rather than to the ref. + release_constraints.publish_constraints(version="3.1.3", ref="v3-1-stable") + + assert triggered[0]["branch"] == "main" + + +def test_workflow_definition_can_be_overridden(triggered): + release_constraints.publish_constraints( + version="3.1.3", ref="v3-1-stable", workflow_branch="my-fix-branch" + ) + + assert triggered[0]["branch"] == "my-fix-branch" + + +def test_nothing_is_dispatched_when_declined(monkeypatch, triggered): + monkeypatch.setattr(release_constraints, "confirm_action", lambda *a, **kw: False) + + release_constraints.publish_constraints(version="3.1.3", ref="v3-1-stable") + + assert triggered == [] + + +@pytest.mark.parametrize( + ("version", "expected_stage"), + [ + pytest.param("3.1.3rc2", "candidate", id="rc-is-a-candidate"), + pytest.param("3.1.3", "final", id="plain-version-is-final"), + ], +) +def test_the_prompt_names_the_stage_the_version_implies(monkeypatch, version, expected_stage): + """The stage is never passed separately, so the operator sees what the version implies.""" + prompts: list[str] = [] + monkeypatch.setattr(release_constraints, "confirm_action", lambda prompt, **kw: prompts.append(prompt)) + monkeypatch.setattr(release_constraints, "trigger_workflow_and_monitor", lambda **kw: None) + + release_constraints.publish_constraints(version=version, ref="v3-1-stable") + + assert expected_stage in prompts[0] diff --git a/dev/breeze/tests/test_run_utils.py b/dev/breeze/tests/test_run_utils.py index 4a9f01355b983..541eb55f987cd 100644 --- a/dev/breeze/tests/test_run_utils.py +++ b/dev/breeze/tests/test_run_utils.py @@ -16,14 +16,20 @@ # under the License. from __future__ import annotations +import socket import stat from unittest import mock +import pytest + +from airflow_breeze.global_constants import SIMPLE_AUTH_MANAGER_VITE_DEV_PORT, VITE_DEV_PORT from airflow_breeze.utils.run_utils import ( + _find_occupied_local_ports, change_directory_permission, change_file_permission, check_if_buildx_plugin_installed, run_command, + run_compile_ui_assets, ) @@ -57,6 +63,68 @@ def test_run_command_dry_run_quiet_does_not_execute(mock_subprocess_run): assert result.stderr == "" +def test_find_occupied_local_ports(): + with socket.socket() as unused_socket: + unused_socket.bind(("127.0.0.1", 0)) + unoccupied_port = str(unused_socket.getsockname()[1]) + + with socket.create_server(("127.0.0.1", 0)) as server: + occupied_port = str(server.getsockname()[1]) + + assert _find_occupied_local_ports((unoccupied_port, occupied_port)) == [occupied_port] + + +@mock.patch("airflow_breeze.utils.run_utils._run_compile_internally") +@mock.patch("airflow_breeze.utils.run_utils._find_occupied_local_ports", return_value=[]) +def test_run_compile_ui_assets_checks_both_dev_ports(mock_find_occupied_ports, mock_run_compile): + result = run_compile_ui_assets( + dev=True, run_in_background=False, force_clean=False, additional_ui_hooks=[] + ) + + assert result == mock_run_compile.return_value + mock_find_occupied_ports.assert_called_once_with((VITE_DEV_PORT, SIMPLE_AUTH_MANAGER_VITE_DEV_PORT)) + + +@pytest.mark.parametrize( + "occupied_ports", + [ + pytest.param([VITE_DEV_PORT], id="airflow-ui"), + pytest.param([SIMPLE_AUTH_MANAGER_VITE_DEV_PORT], id="simple-auth-manager-ui"), + ], +) +@mock.patch("airflow_breeze.utils.run_utils._clean_ui_assets") +@mock.patch("airflow_breeze.utils.run_utils._find_occupied_local_ports") +@mock.patch("airflow_breeze.utils.run_utils.console_print") +def test_run_compile_ui_assets_exits_before_cleanup_when_dev_port_is_occupied( + mock_console_print, mock_find_occupied_ports, mock_clean_ui_assets, occupied_ports +): + mock_find_occupied_ports.return_value = occupied_ports + + with pytest.raises(SystemExit) as ctx: + run_compile_ui_assets(dev=True, run_in_background=False, force_clean=True, additional_ui_hooks=[]) + + assert ctx.value.code == 1 + mock_console_print.assert_called_once_with( + "[error]Cannot start UI development servers because the following local port(s) " + f"are already in use: {', '.join(occupied_ports)}.[/]\n" + "[info]Stop the processes using these ports and try again.[/]" + ) + mock_clean_ui_assets.assert_not_called() + + +@mock.patch("airflow_breeze.utils.run_utils._run_compile_internally") +@mock.patch("airflow_breeze.utils.run_utils._find_occupied_local_ports") +def test_run_compile_ui_assets_does_not_check_dev_ports_for_static_build( + mock_find_occupied_ports, mock_run_compile +): + result = run_compile_ui_assets( + dev=False, run_in_background=False, force_clean=False, additional_ui_hooks=[] + ) + + assert result == mock_run_compile.return_value + mock_find_occupied_ports.assert_not_called() + + @mock.patch("airflow_breeze.utils.run_utils.run_command") @mock.patch("airflow_breeze.utils.run_utils.console_print") def test_check_buildah_is_installed(mock_console_print, mock_run_command): diff --git a/dev/breeze/tests/test_selective_checks.py b/dev/breeze/tests/test_selective_checks.py index a99d5c7bf863d..d516aaeee4ffa 100644 --- a/dev/breeze/tests/test_selective_checks.py +++ b/dev/breeze/tests/test_selective_checks.py @@ -29,6 +29,7 @@ ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS, CI_AMD_PLATFORM, CI_ARM_PLATFORM, + CURRENT_PYTHON_MAJOR_MINOR_VERSIONS, DEFAULT_KUBERNETES_VERSION, DEFAULT_PYTHON_MAJOR_MINOR_VERSION, JAVA_SDK_VERSION, @@ -56,6 +57,9 @@ ALL_KUBERNETES_VERSIONS_AS_LIST = "[" + ", ".join([f"'{v}'" for v in ALLOWED_KUBERNETES_VERSIONS]) + "]" ALL_PYTHON_VERSIONS_AS_STRING = " ".join(ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS) ALL_PYTHON_VERSIONS_AS_LIST = "[" + ", ".join([f"'{v}'" for v in ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS]) + "]" +CURRENT_PYTHON_VERSIONS_AS_LIST = ( + "[" + ", ".join([f"'{v}'" for v in CURRENT_PYTHON_MAJOR_MINOR_VERSIONS]) + "]" +) DEFAULT_HELM_K8S_VERSION = ALLOWED_KUBERNETES_VERSIONS[0].lstrip("v") LAST_HELM_K8S_VERSION = ALLOWED_KUBERNETES_VERSIONS[-1].lstrip("v") @@ -2884,6 +2888,23 @@ def test_no_commit_provided_trigger_full_build_for_any_event_type(github_event): ) +# The image cache is pushed for `python-versions`, so a narrowed list leaves the remaining +# versions with no cache and they build from scratch on every run. `refresh-image-registry-cache.yml` +# forces the label below for exactly that reason; this pins the behaviour it relies on. + + +def test_all_python_versions_with_all_versions_label(): + """Set on the event that would otherwise narrow: a text-only push.""" + stderr = SelectiveChecks( + files=("INTHEWILD.md",), + commit_ref=NEUTRAL_COMMIT, + github_event=GithubEvents.PUSH, + pr_labels=("all versions",), + default_branch="main", + ) + assert_outputs_are_printed({"python-versions": CURRENT_PYTHON_VERSIONS_AS_LIST}, str(stderr)) + + @pytest.mark.parametrize( "github_event", [ diff --git a/dev/breeze/uv.lock b/dev/breeze/uv.lock index 4c45cf8fe6edf..5124566ec0376 100644 --- a/dev/breeze/uv.lock +++ b/dev/breeze/uv.lock @@ -69,8 +69,8 @@ requires-dist = [ { name = "boto3", specifier = ">=1.37.2" }, { name = "click", specifier = ">=8.3.0" }, { name = "filelock", specifier = ">=3.13.0" }, - { name = "flit", specifier = ">=3.12.0" }, - { name = "flit-core", specifier = ">=3.12.0" }, + { name = "flit", specifier = ">=3.12.0,<4" }, + { name = "flit-core", specifier = ">=3.12.0,<4" }, { name = "gitpython", specifier = ">=3.1.40" }, { name = "google-api-python-client", specifier = ">=2.142.0" }, { name = "google-auth-httplib2", specifier = ">=0.2.0" }, @@ -254,30 +254,30 @@ wheels = [ [[package]] name = "boto3" -version = "1.43.57" +version = "1.43.62" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "botocore" }, { name = "jmespath" }, { name = "s3transfer" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/c7/30/324319a914752e4021358ccf8252c04364eb8358f9d41d9c3f021959b363/boto3-1.43.57.tar.gz", hash = "sha256:549c95e45f9b04cf0c69727632dbdda23b84dcd3d0ed7981c6aaff72ef9cb5af", size = 112690, upload-time = "2026-07-27T19:31:09.32Z" } +sdist = { url = "https://files.pythonhosted.org/packages/b9/c7/f7732c5e1abf7270a6bbbce47338d25ea66a30df658cffd1d17bb5f735fb/boto3-1.43.62.tar.gz", hash = "sha256:0bf920e0739346e81c7310b685a3f783bf1fcc62ce7d5c7016508fa25c0d261f", size = 112668, upload-time = "2026-07-31T19:35:17.257Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/80/a4/2c4ee25556a997a81ea01073962cf8351da3707412c584d053b3861d09e8/boto3-1.43.57-py3-none-any.whl", hash = "sha256:115ed9cac409d9b57e2437b52fdb4286660d12a2bd44a0f48d530e68623d09a7", size = 140028, upload-time = "2026-07-27T19:31:07.226Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f0/5e1a392c817e395b140c18c12a00c0c65c69f8d63da26ad4387aebf2172b/boto3-1.43.62-py3-none-any.whl", hash = "sha256:0bb298e7ffd72b91615df44bf71c417df80a29d844971e5d665b8bd743a4bb35", size = 140025, upload-time = "2026-07-31T19:35:15.347Z" }, ] [[package]] name = "botocore" -version = "1.43.57" +version = "1.43.62" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "jmespath" }, { name = "python-dateutil" }, { name = "urllib3" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/0b/92/1f8a454cf90bcb0c0e32a25817441cb7f3702fdd76710d7018abad12a2fc/botocore-1.43.57.tar.gz", hash = "sha256:001a5653bebc03b862bde2da63bad4adb0b30072a3f247aba4daae5aa097b546", size = 15739550, upload-time = "2026-07-27T19:30:57.916Z" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/8d/36af6d99269a701f83809b87a01f4728699eb825ebdedee3a3d515b18f61/botocore-1.43.62.tar.gz", hash = "sha256:94efc419c9f0f41dc2415e4b6b62f04ae21b3ce3930fac47214c4d3f361ea8b8", size = 15818261, upload-time = "2026-07-31T19:35:06.235Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/41/7a/1cfe9c296df0fa6e0143c8622b10f21fa8fb9cce25450e9a8e5cf6523e4b/botocore-1.43.57-py3-none-any.whl", hash = "sha256:319c6d79f66c3f3f2b538f7dcdc90e410a15a7bfced1db06479134947e629482", size = 15424433, upload-time = "2026-07-27T19:30:55.08Z" }, + { url = "https://files.pythonhosted.org/packages/c0/65/d5dae96de68ffc55acf87c3bae76e9dabeeca92aadf1223f20e9a7860aef/botocore-1.43.62-py3-none-any.whl", hash = "sha256:76de153de1ba3e242b2e6df6a13ab8a3fb35d17db562462969e661457b63166e", size = 15502622, upload-time = "2026-07-31T19:35:02.697Z" }, ] [[package]] @@ -509,59 +509,59 @@ wheels = [ [[package]] name = "cryptography" -version = "49.0.0" +version = "50.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, { name = "typing-extensions", marker = "python_full_version < '3.11'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/1f/99/d1c90d6041656cc6ee229dc99cd67fd0cd5aec3c5f7d72fffc27cc750054/cryptography-49.0.0.tar.gz", hash = "sha256:f89660a348f4f78a92366240a61404e337586ef7f5909a2fef59ca88ef505493", size = 854345, upload-time = "2026-06-12T20:02:30.512Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9b/22/adf66990e63584a68dfb50c24f48a125c07b1699899381c8151e63ed458c/cryptography-49.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:966fe0e9c67490071f14c0d2b1cb2dfb3023c5ce39457343931415f08382f2db", size = 4032100, upload-time = "2026-06-12T20:02:32.143Z" }, - { url = "https://files.pythonhosted.org/packages/09/41/3797cfaf69cae04a13ee78ebd83f0678d9c02b4779d21ce24445326f1a69/cryptography-49.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:36d1709f992593689b45bda411498d62c6e365f2ca00b84657d4dadd24de16db", size = 4692978, upload-time = "2026-06-12T20:01:21.305Z" }, - { url = "https://files.pythonhosted.org/packages/e6/8b/43011f7ebe515a8aa20d61f290a326cd890c2e738e16e59eaff8d9c3a412/cryptography-49.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0e959b578856a3924bc0cbb710fc12c387b9412a951389f3ca61704a9e25f325", size = 4716422, upload-time = "2026-06-12T20:01:48.566Z" }, - { url = "https://files.pythonhosted.org/packages/4a/91/01ce7303a4579e6d3a6abef01bd322848e9ea7a219adcabc5048b9033571/cryptography-49.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:53ecee2e23f7169b6117e99fc8a944e5e50f79e69758a83b52a00cb98ab2b2d2", size = 4700503, upload-time = "2026-06-12T20:02:47.091Z" }, - { url = "https://files.pythonhosted.org/packages/62/99/a2c95cf8293f07491e9e27c20cc4dcd18176d944e674679adeb1d0173fd6/cryptography-49.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2eda353d8a27bcbcaa4cbed18994a74ab4d19a2ca897db188ea269ab9b71419b", size = 5309779, upload-time = "2026-06-12T20:02:08.987Z" }, - { url = "https://files.pythonhosted.org/packages/20/2c/0622f20ff02b2ef32558733443805dc82fd4c275be01b2d19d14676f3a1b/cryptography-49.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2afe9051da7ae7bd5905da5a949280c7d2bb75682e188f650a9d0f2756b834c6", size = 4749683, upload-time = "2026-06-12T20:02:03.335Z" }, - { url = "https://files.pythonhosted.org/packages/a3/5b/c5246635d5fd3b64e0d45ae10e99fd32fe9676a79915ccfe5a61ba9af1a5/cryptography-49.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:0b82e28ee398a386f0807bba7884d30f25218855690f45115831bcce5d90822c", size = 4337874, upload-time = "2026-06-12T20:02:54.323Z" }, - { url = "https://files.pythonhosted.org/packages/6d/88/05563c7fe2e914e87d1a536d06fe83e66b4e1d95cb593e05aea375531da8/cryptography-49.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:ccac2bfebc306b862133e3bb71f3f6ee8bb525240089b2d952e4144b3a6d5da7", size = 4700283, upload-time = "2026-06-12T20:01:34.822Z" }, - { url = "https://files.pythonhosted.org/packages/c4/b6/d7696e4e890d6ae1469935164c9e5215c557671cb78d6e3f458ccceaa632/cryptography-49.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:d0527ce944105f257f605a827d6ebead966c752038b6e8656abb9c5edee6fc68", size = 5265844, upload-time = "2026-06-12T20:01:24.09Z" }, - { url = "https://files.pythonhosted.org/packages/a9/3c/f3ad17eecc1a57b0ba236dc01f90e783c51f4a2f35f64777cc4f47a184b2/cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:cbc77da8c523d5abd028635ba850a6966fcee2c82e2bf65a41d1d8afe0f98be9", size = 4749290, upload-time = "2026-06-12T20:01:30.848Z" }, - { url = "https://files.pythonhosted.org/packages/4f/01/339573cf1023163a400b0b5d16f6d507de413b9f60be6fd1b77feeaf6737/cryptography-49.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b87e65d263b3e5d3bb92a57e2a6638e2f31110fa7aa890c7b2dbba42248d0a3f", size = 4834612, upload-time = "2026-06-12T20:01:29.246Z" }, - { url = "https://files.pythonhosted.org/packages/71/fd/577302e213a1be9468f92d1afef66fcf1ef83d516819d9992ca547f592bd/cryptography-49.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:66ec79c3904820572d7e987abdf304281f141d37ad9a489b8e97066e7b9b6459", size = 4980804, upload-time = "2026-06-12T20:01:42.853Z" }, - { url = "https://files.pythonhosted.org/packages/1f/09/f42b1d190c5ba75f72062a387f8030d1d75f6ab035788f1d9c4b01de6525/cryptography-49.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:e5dfc1e64de5677cec922ffa8da89c546d0415bf6efdf081842e5d44c84e1f0e", size = 3810026, upload-time = "2026-06-12T20:02:39.262Z" }, - { url = "https://files.pythonhosted.org/packages/ec/9e/db72b3ae7fc9cfad53e630e56c6ae83b9b6ff0bf3718ffb8012d20b3aabf/cryptography-49.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:73a205dce83953d131a4aa1e0fd917a2fd1c5b1eef251e9d7152efefcbf5caf7", size = 4013892, upload-time = "2026-06-12T20:02:10.735Z" }, - { url = "https://files.pythonhosted.org/packages/86/12/c48a424f38db03027be9f7ed5c7dc5de9933dbee992865f98b13727a009d/cryptography-49.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:196ecd6a36e4e9aa10270393bb98d8df88fccee0bf1e5128b91ae4eb4375896d", size = 4678835, upload-time = "2026-06-12T20:02:48.743Z" }, - { url = "https://files.pythonhosted.org/packages/68/28/8a3ad4653662c93fc44dc4e5d8fd374c25c42e07b34bbfbadf49cf57a5a8/cryptography-49.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7abcee80084cda3f7691f3eb1ce480d8df49cec637b429aa35986c1de71738aa", size = 4697239, upload-time = "2026-06-12T20:02:56.03Z" }, - { url = "https://files.pythonhosted.org/packages/a8/b2/2193fc74f81aee4f9b62733133b73b5176718932ed8f2e4b03fa040480a6/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:4ae387c9cb68ea569ca17e490d66d8142b81c3cc814bf179974b7d146e490bbb", size = 4685593, upload-time = "2026-06-12T20:02:50.666Z" }, - { url = "https://files.pythonhosted.org/packages/47/f1/1d3eaa243bfc5de4a187b22aa8c048b3e4980bfbe830ac46e6bac2e66947/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:f37d847238971164fdbc68ade6f6574aecc9c0af714190e2083429ff68f4ce9d", size = 5289961, upload-time = "2026-06-12T20:01:46.468Z" }, - { url = "https://files.pythonhosted.org/packages/58/39/2d51306721330c486495853eda1c567880ff036de15a14c4b74f399934af/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:c2bc30226390d60ea19d9f82b19db005fe0452154a23c1c410c12ea801e43561", size = 4731145, upload-time = "2026-06-12T20:02:16.832Z" }, - { url = "https://files.pythonhosted.org/packages/17/50/983e838c7fd0d87fd8c969bcdd328edaf5f756e38df5281637424c155873/cryptography-49.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:07cab27cc7b7e0fd28e5e26bb9eeedde5c135c868b46de4a27845abe94af6122", size = 4321719, upload-time = "2026-06-12T20:02:52.611Z" }, - { url = "https://files.pythonhosted.org/packages/a7/f5/8f571d7e27c55bce9f76f026143bcb1e040a4233149ecca0bea5fa5dd5f7/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:b20133d204d2bb56ba047642199603876c872026ca53e79c35b83772ab2cc505", size = 4685209, upload-time = "2026-06-12T20:02:07.282Z" }, - { url = "https://files.pythonhosted.org/packages/e7/84/0e27016a6fc5a0886f797018b26aa42f40c09a82332bff77822a451deaaa/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:b970c6da94d5bb18629db453d14f2a1300f6bf59b61e9b82377931ef95504866", size = 5246285, upload-time = "2026-06-12T20:01:32.439Z" }, - { url = "https://files.pythonhosted.org/packages/11/2d/5e1fb307cb5931881516b464c98774b3f2c36b5d4bb9a2830253cf553cad/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:d8ecde755e2e91bf773fc94e8c9d730cd7f2007004cb492263a794ec3899a1c8", size = 4730441, upload-time = "2026-06-12T20:02:01.469Z" }, - { url = "https://files.pythonhosted.org/packages/e4/c0/bff5a02ee731d207d6a1ed51732549d8c53d2bc8da1d10ec6f2844201d68/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e3fb64c420688e5319ae25113a354015abbd8dffbfbc41781a1ea66fc7622ac3", size = 4815869, upload-time = "2026-06-12T20:01:36.574Z" }, - { url = "https://files.pythonhosted.org/packages/b9/26/814681d14248d95d73d5c3eea0c39a94eb8302df966f670a2c60de90974b/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32703d93296f5c1f4b53349ad3a250c2cae0fdecd3a3dd5d47e616d8d616af27", size = 4960948, upload-time = "2026-06-12T20:02:18.688Z" }, - { url = "https://files.pythonhosted.org/packages/4c/fe/93ecac273d3738939d023612ad12cca9a3740a5345d69fda04134c43fd96/cryptography-49.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:33cd0565932807baddb67b96dbee92f2c374b5c89dee09fd74079aeb8c8dba61", size = 3799153, upload-time = "2026-06-12T20:01:39.059Z" }, - { url = "https://files.pythonhosted.org/packages/19/2a/5bb823f5bedcf80718cea7fbc95ec5515cca3769633c4b01a32be7f30e7c/cryptography-49.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ec5e529fb80935c94fe7b729f9972b50e351a0e6b50aa294fd5cabb109fcc29a", size = 4025947, upload-time = "2026-06-12T20:01:25.745Z" }, - { url = "https://files.pythonhosted.org/packages/3d/df/40577043ca124e17012f408ddddaeb213b856336ac82ddb3bc915f39e29f/cryptography-49.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f78ff2c9ed8dc2d036b0f4d640e22522213d047c1b14e61205a7e55c80a494d4", size = 4692429, upload-time = "2026-06-12T20:01:53.628Z" }, - { url = "https://files.pythonhosted.org/packages/2c/99/2d13299eb3dd27b02dcfaafcc91d6b5cb3329f7cbd6d8f51921acd566c1a/cryptography-49.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:35b151772baff2c74cba7fa290ceaff4c3b11c0c881eb93eb5dbc05a7cfbba18", size = 4700968, upload-time = "2026-06-12T20:02:45.383Z" }, - { url = "https://files.pythonhosted.org/packages/a5/4d/9c0cd02f95e2602dd5e563da149ee0830abef3537be8b34dc56281ebe27a/cryptography-49.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0f21641cf4b30fca7aee061ced0ec7ad7b073518088b7c9969a297c0ae796c69", size = 4697758, upload-time = "2026-06-12T20:01:41.13Z" }, - { url = "https://files.pythonhosted.org/packages/24/01/186c825898477d77e2324d5360fefe622ff1d8d1963ec0554e2cada8ec77/cryptography-49.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:9e82dcc8e56052715fb18b2429e3bca4823b1629136a2084fc45a9a5cecb9b64", size = 5298863, upload-time = "2026-06-12T20:02:24.579Z" }, - { url = "https://files.pythonhosted.org/packages/b8/7b/62cbbab75d0659865bf0273790031544a0b16c8072d258f9428dcd8190dc/cryptography-49.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:6f2debedf9ca60cf1d5bd466475638af5130f89965605cd818484d19987d3a21", size = 4735983, upload-time = "2026-06-12T20:01:50.14Z" }, - { url = "https://files.pythonhosted.org/packages/6c/72/3e798c064bc39e471008075d0f9bc9daf77a80879c092e4a8e170c585ed4/cryptography-49.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:8c25ceb16df5b9435f3f6a9829204985b0e0cbee3b48aacd432c7d2c850b44d9", size = 4334173, upload-time = "2026-06-12T20:01:44.743Z" }, - { url = "https://files.pythonhosted.org/packages/f0/ee/6fca21d1ac73e06f8bef71940abfd4d2f6472b4bca284d770f32bd4086f6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:28d8b15e6275f12c8a207dc309dfa957903c927d08d0cc937ee3f63f200693cc", size = 4697298, upload-time = "2026-06-12T20:02:20.918Z" }, - { url = "https://files.pythonhosted.org/packages/67/d0/a5fcd3515f0bae49a7b6d0413cc1bdccdcc1fc0047037a0d480642cdc5d6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6fc361c34fb6aac015ce19435876635e5c6d21db31998b0920f675f131e043b8", size = 5254338, upload-time = "2026-06-12T20:02:22.737Z" }, - { url = "https://files.pythonhosted.org/packages/a0/84/84fe36f19caf857d61cb7fc9c63035a47ffabd84ea12d1d393148efa3615/cryptography-49.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:2400ef9c9e2299a25614eb1dea3db54a69b1349efd043bfac9c67630d136df36", size = 4735650, upload-time = "2026-06-12T20:02:41.389Z" }, - { url = "https://files.pythonhosted.org/packages/6c/a0/db537264e234f7273a73ec020873d6d6b39dfd8a53db78b550ca8320440e/cryptography-49.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:67e1d20ad9ef3a563c59ef22e7a8a0b8210bd26604369ea4a30a7c66aefe504e", size = 4834820, upload-time = "2026-06-12T20:01:51.847Z" }, - { url = "https://files.pythonhosted.org/packages/93/77/8df9eb486495979bccecd1062e2eaf435250e84437040295b57d09048b0b/cryptography-49.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:42b0684e0e40cf26122427802486f6d93aea593612603a94fbf260c7eb1e9c1b", size = 4967968, upload-time = "2026-06-12T20:02:12.524Z" }, - { url = "https://files.pythonhosted.org/packages/c2/e6/f60198ea8d9dfa15fff9ed4ca02ce362f6eadd9ba757dcc50634c4257b63/cryptography-49.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:026ac7423e6fa66872d3bf889be5974507da3944f866f704fa200eadacd00001", size = 3785547, upload-time = "2026-06-12T20:02:26.847Z" }, - { url = "https://files.pythonhosted.org/packages/63/d3/4a83af35d65e3fad632c926fad684c193ea4398569ccb0bbbc7fe8f5dc9a/cryptography-49.0.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:fc1e275c2f1d97b1a6450b8b0ea3ebfa6e087a611c2b26cb2404d48588abab7b", size = 3993685, upload-time = "2026-06-12T20:02:14.883Z" }, - { url = "https://files.pythonhosted.org/packages/d6/a7/f9dac0ab7f80368c56993a7bf638ef9935f825c91902798481fac0898138/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:c83782480a4a9da4d0feb51950131ba32e12e70813848b3343f6e18c28a66838", size = 4676239, upload-time = "2026-06-12T20:02:28.793Z" }, - { url = "https://files.pythonhosted.org/packages/d7/70/2ba3769dd0ae167e2f33dfa9592d45db6ff9a61d62ca1a5b3d1bdd09068f/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b39efa323140595abd3ecca8529d321ae50f55f3aa3ba9cc81ea56a6011953d5", size = 4715584, upload-time = "2026-06-12T20:01:27.495Z" }, - { url = "https://files.pythonhosted.org/packages/94/64/2923570ac1c0bd3a737aa366ac3abbbbde273042308b8cde95e2364a6e6a/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:b47db11c2c3525083296069b98ac5221907455e989ae0c2e3008bde851921615", size = 4675885, upload-time = "2026-06-12T20:01:55.49Z" }, - { url = "https://files.pythonhosted.org/packages/ab/f8/614dc7e051418cfe53d55173c1e24c6b0085e89996fe90508c2fdf769aef/cryptography-49.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:084ef1af862eb07ec46d25f68689f2102a9fc0e05ce7b80f14f5fe51e4eef0f6", size = 4715449, upload-time = "2026-06-12T20:02:05.469Z" }, - { url = "https://files.pythonhosted.org/packages/aa/50/a9caea39ad19c431c1a3f8a31114df65b260cdfe67786b6c7e7c040c4c44/cryptography-49.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:be9fcb48a55f023493482827d4f459bd263cc20efde64f204b97c123201850c6", size = 3783731, upload-time = "2026-06-12T20:02:43.319Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/de/41/6cbdcf9142d00fe82836fbb51e503e58088575cf7a0fe1dbff6695bf0840/cryptography-50.0.0.tar.gz", hash = "sha256:eeac2acb5a20ed25e0ad6d1df9891a520b78b404266b6d11778f25d5d691a6c9", size = 880201, upload-time = "2026-07-31T14:25:10.11Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c5/5c/59086b4aac5e879d38ddbcf74e4be7ade89cebc3eb199a55da998c3bb46a/cryptography-50.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:031e2d5dd4bb9caa3ca9c82e5a197fd8ae680232cee62603d1a813f3f07e3d03", size = 4001252, upload-time = "2026-07-31T14:23:33.331Z" }, + { url = "https://files.pythonhosted.org/packages/57/ef/8f2df13c7216bcad3e1c74e07f6e193d93e998e114f524a53877c9af27ad/cryptography-50.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fd9192b7b70c573d7f214eb1ae35e00d359f6f5e4b27c7e21e30de1fc6204645", size = 4719554, upload-time = "2026-07-31T14:23:35.611Z" }, + { url = "https://files.pythonhosted.org/packages/d9/41/029086c34d91052fc3b88bcc8056f709a7c915c7a23b235a54eb800b1c97/cryptography-50.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:06a32a980526a6ab9a4b9bf8f7385800791e2bb960903cb6b530e4817509a3b7", size = 4702130, upload-time = "2026-07-31T14:23:37.635Z" }, + { url = "https://files.pythonhosted.org/packages/7d/ff/b6ce0954962e7f7b969f850a883744197bb3910bdfd7b6da162eab7d9f68/cryptography-50.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a1b30560f2acc95aa8b2e06e716a13dbfc97314747b80d9707e307f77b40d6b3", size = 4725244, upload-time = "2026-07-31T14:23:39.471Z" }, + { url = "https://files.pythonhosted.org/packages/06/1e/63a1027cb7fec360a182208e1b7767d5aa1fe57be3d6aa856e69a321edc0/cryptography-50.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:8d89f3976b10b4ce31118de72329025f70d2c6ead14a8217c5514dd2c6d5a78f", size = 5342265, upload-time = "2026-07-31T14:23:41.286Z" }, + { url = "https://files.pythonhosted.org/packages/6b/72/a1116d683a6d7ece94590013882515de087edf9ef0e6292aae615a44df73/cryptography-50.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:b42a28c1844fd9de8f3f7d540e36b66f3a9c83fceac7170ebc7a6a19edd9dcae", size = 4734609, upload-time = "2026-07-31T14:23:43.139Z" }, + { url = "https://files.pythonhosted.org/packages/15/37/36a9c479bbe49acea2636c7fd3360d20f7b7e079c300352011c44850b181/cryptography-50.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:900131fafd8aead39ac7dd3a7e833be754c17a95cfd91221636949fe4eb0aa8a", size = 4356517, upload-time = "2026-07-31T14:23:44.939Z" }, + { url = "https://files.pythonhosted.org/packages/32/98/8a151d64367204cbc63ec65d37502f1d9c53cf4bfc6ec3c532614dbec60d/cryptography-50.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:07949c449a1abcf60d1ee6e88956d89404c7df3c8258f46589e912988e551987", size = 4724529, upload-time = "2026-07-31T14:23:46.93Z" }, + { url = "https://files.pythonhosted.org/packages/22/f6/ec13b470172126464a86bf54d2294a46d29837fc51ba3e45d4047946fb5e/cryptography-50.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:f89831ef99dd7dd169ab06d63a831adb9e20a87aac6d380266bbda5823349169", size = 5299852, upload-time = "2026-07-31T14:23:48.851Z" }, + { url = "https://files.pythonhosted.org/packages/da/3a/f05e32c99d440c9bb891ea0e36c9091891e36be5a9a87ab2ee6ea20729f6/cryptography-50.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:82148ec5bddac30b51a5b3c1945075f896fa022cb93f8e4a01e9f6ee95292c5f", size = 4734462, upload-time = "2026-07-31T14:23:50.861Z" }, + { url = "https://files.pythonhosted.org/packages/ca/dc/bd72b26be8953f80625f63151efd38eee71c76ca6cf591c08ff34615a79e/cryptography-50.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1489e263a8048bb8b6a8bac662eb2d402ea5d2b7b4699b72f385f1e2772db105", size = 4852708, upload-time = "2026-07-31T14:23:52.715Z" }, + { url = "https://files.pythonhosted.org/packages/27/20/c930314a2ab476d15dec966ec87e2e9637bb02b06106b12c0396c57bb603/cryptography-50.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7cec5b856506da6defb290f30c9ee687d5f5e8cb0bd3f6459dde43b0b4fa40ef", size = 5004179, upload-time = "2026-07-31T14:23:54.887Z" }, + { url = "https://files.pythonhosted.org/packages/32/2e/c9db68a0c4bfa28e310707527c0ee3a2bd254104d2e02e68f368e197aa4c/cryptography-50.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30", size = 3840395, upload-time = "2026-07-31T14:23:56.677Z" }, + { url = "https://files.pythonhosted.org/packages/c3/fb/951032a3bf22a5697c83183fb6294a4843772947a70e616c57b3ff5f522e/cryptography-50.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:49e7d93abdbd2990caced757e5fade25302f719c3c8fb6e6fff2dde98999fc41", size = 3989258, upload-time = "2026-07-31T14:23:58.881Z" }, + { url = "https://files.pythonhosted.org/packages/d4/67/91eb047e69c5e845f2f14b8a2e4a1aab0f283cb885531e9e22c8adb176bc/cryptography-50.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:19736989797678c6af1e55cd49055cdbcb55d8f6b5583ac5335f933aba9101dc", size = 4700648, upload-time = "2026-07-31T14:24:00.702Z" }, + { url = "https://files.pythonhosted.org/packages/30/82/85f0f7425c856b9f96459411eb12e74ef72df9caf6f8f15bf23a33ff131f/cryptography-50.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80b63928fa35083b33966ce1efb70e5b9607181e49dcd1c22c8c005e319f667f", size = 4682442, upload-time = "2026-07-31T14:24:02.538Z" }, + { url = "https://files.pythonhosted.org/packages/1a/28/b555a365adff1cca2fbe7b9e487d68a40de6bc67ff2cb587473eb43de0e7/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:d58c3db7cd6eed54e6c06744db55456b65ebd7492ddeae9c1e93cfca7aa857d3", size = 4707596, upload-time = "2026-07-31T14:24:04.394Z" }, + { url = "https://files.pythonhosted.org/packages/72/d8/f52538140cc719df62a01cf87d1c7142318d235817109d6f4054d7c352d6/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:df2a58a472f332225671c35b0a830208b86d004f82baa8530fa3782c85646533", size = 5314552, upload-time = "2026-07-31T14:24:06.31Z" }, + { url = "https://files.pythonhosted.org/packages/38/14/6120e5bd7c5aa022ad15424ba4d5c5269d0d9448ed4d55e492ea91e3c1c4/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:11b74db56cdbe3cdee6e3f6982ecb70334fa10dce99ed58bf7894aaaa3b2a037", size = 4717113, upload-time = "2026-07-31T14:24:08.349Z" }, + { url = "https://files.pythonhosted.org/packages/fa/71/190bf38c3ee2e0f8efc9860ae100c9df4169742eef274b91e7aa1cb133b9/cryptography-50.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f59e38625469987d7ef6d495323c55e7db6c212eaf6112267e0d3b565a2e9c9f", size = 4338580, upload-time = "2026-07-31T14:24:10.227Z" }, + { url = "https://files.pythonhosted.org/packages/3a/63/504ccfbbe61fd8aa983f7f146399cdf034c72c2fc55f5b2dfdcdcdb20c99/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:ecfed7367f965a0328cfbdd70da860f15441f002f613185668c6e6ebf5a0ac11", size = 4707038, upload-time = "2026-07-31T14:24:12.169Z" }, + { url = "https://files.pythonhosted.org/packages/01/77/2cf79bbfc4d12ca106437a6e170d6aaa01a373e93093118aaaef0e801bd4/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:9aa87839c383bdbab6ef865787a1fb877af8dd03464c4400322726feaaadfc6d", size = 5273110, upload-time = "2026-07-31T14:24:14.38Z" }, + { url = "https://files.pythonhosted.org/packages/e5/45/8aae2972c520145377ea3559a605a899bebe227bf070b33cdb445929a9b9/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:6ba6a53445bd3cfa809ef3ef5f1589aa6ba08784a1d962bf47d0940e871dab1c", size = 4716439, upload-time = "2026-07-31T14:24:16.415Z" }, + { url = "https://files.pythonhosted.org/packages/7b/20/4fe50b619a48c2525cc46e2dbc1ac490708d704be5d467bdaac6dc955682/cryptography-50.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3f5735ffe4996d28b809371756219f5354864902a3b9e7c0b9ee87041209fc9c", size = 4837383, upload-time = "2026-07-31T14:24:18.553Z" }, + { url = "https://files.pythonhosted.org/packages/92/91/3a31366e183343d3703f8995c095f5734676bd6938118047e50fcf279eb4/cryptography-50.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1b4a266766514614f8aa60416e71f2fc6e575d36e7bdc90f644fadb2f4b75b95", size = 4985772, upload-time = "2026-07-31T14:24:20.385Z" }, + { url = "https://files.pythonhosted.org/packages/74/9a/02ffe35b2853d121689871eb5dce862092562b3a1ed5cc98f1aaed441506/cryptography-50.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:12b9c6996425c76ea6c457ace4f3073e715b8c545add07cd1a8f3a4f90691269", size = 3816291, upload-time = "2026-07-31T14:24:22.125Z" }, + { url = "https://files.pythonhosted.org/packages/03/37/73d005be173aff344af30e9fd2a576575cb2391a7101d9cd3842e1fa8cce/cryptography-50.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ccdc4a71a4dabae05de219404f9f4abc38e3b58422177ff93d0da05967dafa07", size = 4036009, upload-time = "2026-07-31T14:24:24.122Z" }, + { url = "https://files.pythonhosted.org/packages/ff/c6/7a6202a534e32103a285b7834a120869557fe198d51d7cfe59754c8bda9c/cryptography-50.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:910e1d2668e7de9648f2bcee30e180db2a6b15c30f887d7c4c93ddf96e3992e3", size = 4745252, upload-time = "2026-07-31T14:24:26.118Z" }, + { url = "https://files.pythonhosted.org/packages/85/4f/0fa8c2f4428198f15d9ff8d63400e27afbf94ce833f6108da1eb3753f945/cryptography-50.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a91296cb61e8df6f86d0c19cc4068228da256bf59bf86049fbd821084565327f", size = 4728939, upload-time = "2026-07-31T14:24:27.994Z" }, + { url = "https://files.pythonhosted.org/packages/d1/63/54dd723490ba2dc09b299682c10b38db38f159728bcaae8c591b8af2f22d/cryptography-50.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e722f16708d854fe924790e051061f6704a472c3bac347b6fd88033ea8dd0dc5", size = 4748483, upload-time = "2026-07-31T14:24:30.254Z" }, + { url = "https://files.pythonhosted.org/packages/1d/dd/7c77d26285cc7f6991efce64a0f5b4f9383bfa5dd8c5033003eaf7db4cdb/cryptography-50.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d764dcf130c428ef66786f866dd750f53182bc608813489915e9fc106bb0c82f", size = 5367599, upload-time = "2026-07-31T14:24:32.457Z" }, + { url = "https://files.pythonhosted.org/packages/46/c9/f60aed34c013f317f92817b6c171c2d22a78270fa41109bd4b08af26b194/cryptography-50.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:105110f43a471dbd0060b9c9516cb8a6a79233631a04cc2ba16f28323ac6e025", size = 4762647, upload-time = "2026-07-31T14:24:34.599Z" }, + { url = "https://files.pythonhosted.org/packages/be/f3/f9a0173b139372c3a48ed98154b45cc6b9de17c789d5ab552e621c293609/cryptography-50.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:828743d939e9629bc267b8e2d08d8bb67cd4319c771a33d4b18b22dd8fb7440a", size = 4385197, upload-time = "2026-07-31T14:24:36.647Z" }, + { url = "https://files.pythonhosted.org/packages/d8/36/83bb81f6e569bc38e1e4a7bc80f29b46bb9601920bc455fc8e888f5d5742/cryptography-50.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:2a8183b489dc1f7f80f135780fadc1108f14b31b8a40411c7a5b17425f65f28b", size = 4748095, upload-time = "2026-07-31T14:24:39.493Z" }, + { url = "https://files.pythonhosted.org/packages/6b/16/d3008eff98c764979865834c3d386d4fd041b5f52e7f34fc29ac1a5eb515/cryptography-50.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6e7d61120573a7f2cd94cc095f9e81f6967c61ccdf194285aa143ecec8e0b708", size = 5325948, upload-time = "2026-07-31T14:24:41.556Z" }, + { url = "https://files.pythonhosted.org/packages/9c/f8/d97f9603efda3888187bfdb893f26c41be4735c10631d05d284ee6b047c4/cryptography-50.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:37fdb0d0111f1e2ff07139dfb79f1b49531f8e213c46f1163dd7642979b58c47", size = 4762400, upload-time = "2026-07-31T14:24:43.636Z" }, + { url = "https://files.pythonhosted.org/packages/64/a2/4615c8f7d81a00b1d6e6afe19f694e1543582349fb5f4076f6cb5dc36485/cryptography-50.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c87f62a3d3b9888ed0fdde100ec06aa61ca9cd44bad9057d1dff9a516b5f5bb9", size = 4878208, upload-time = "2026-07-31T14:24:45.522Z" }, + { url = "https://files.pythonhosted.org/packages/d2/1a/efcfb02f91407149a0dacffffab791f7e19bf6385f63b3666dc8b5e5c9c8/cryptography-50.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:65c2c3add92b45fd0709db8594536aea39c2a67af0e27ffcf049c498501140b7", size = 5037050, upload-time = "2026-07-31T14:24:47.697Z" }, + { url = "https://files.pythonhosted.org/packages/57/30/4a22984d4f1bdfb8c054f07a92bc176b97a3134cc1d6c4b3bffb1f3688b4/cryptography-50.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:d24fead1d4d076e1bfb006dcec392074a3cd8d7b4fc8a595aa64073b2b7a96ba", size = 3874135, upload-time = "2026-07-31T14:24:50.085Z" }, + { url = "https://files.pythonhosted.org/packages/9d/3e/e54cde8c01631a5a8226ccd617eab9e57fd5cfdad90f1a9e6bb570794631/cryptography-50.0.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:5e34edd123674534acd70147f0ca331eaa2c74e6325fb2028c886aa26ba0b68c", size = 3963170, upload-time = "2026-07-31T14:24:51.968Z" }, + { url = "https://files.pythonhosted.org/packages/01/b6/0b9e125e90f3d2dcf599a218a899cda7326a3158cfa258723f0b398b08f6/cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:8eb5e1172eb569ea8a872796576e6a67c276351728b6455d5beb01242b027c6a", size = 4692441, upload-time = "2026-07-31T14:24:53.743Z" }, + { url = "https://files.pythonhosted.org/packages/53/c9/a5151588710785a96d7bc4de27d4cd62f263bbbcb203cfe29df537eb6505/cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:910d11e1a385c654bf738bf3e6b8e6ed5de0f5610fcae2be9e5b398d8081d20e", size = 4699810, upload-time = "2026-07-31T14:24:55.746Z" }, + { url = "https://files.pythonhosted.org/packages/c7/1a/15b92b25eb6ce3089cd49377ae990a0f3ad485a510f968aed1f19dbdcdf2/cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:62598a8a57f815db4c6259a4e97d857dab56697e7de8e8ab02352ab74da1995d", size = 4691924, upload-time = "2026-07-31T14:24:58.082Z" }, + { url = "https://files.pythonhosted.org/packages/62/15/219075012ab13e8905f3cd572204f4acb4b111df787104346b9bc0cea789/cryptography-50.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:07479a1cb08219ab719147e742e76090c9c773321959bb94946fffdd397a6437", size = 4699593, upload-time = "2026-07-31T14:24:59.951Z" }, + { url = "https://files.pythonhosted.org/packages/8e/b5/c2c5fce26f0ee40d21bafe7f191d29a34b35a65ac4fe8a1191d1983612e9/cryptography-50.0.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c99c003e088647b8a5b7c145d6f78c335f6348332b62e142d411c4b63d1460b9", size = 3813796, upload-time = "2026-07-31T14:25:02.298Z" }, ] [[package]] @@ -596,7 +596,7 @@ name = "exceptiongroup" version = "1.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "typing-extensions", marker = "python_full_version < '3.11'" }, + { name = "typing-extensions" }, ] sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" } wheels = [ @@ -614,11 +614,11 @@ wheels = [ [[package]] name = "filelock" -version = "3.32.0" +version = "3.32.2" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/c0/80/8232b582c4b318b817cf1274ba74976b07b34d35ef439b3eb948f98645a1/filelock-3.32.0.tar.gz", hash = "sha256:7be2ad23a14607ccc71808e68fe30848aeace7058ace17852f68e2a68e310402", size = 213757, upload-time = "2026-07-21T13:17:42.898Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/57/3ba6e6cb097f85b855b00163d169f35365f44277df044dcf96d55b8f62a3/filelock-3.32.2.tar.gz", hash = "sha256:c33351e1f49cae33414acbc6d56784e6ecee82514ec90795da1161fc4836b5b8", size = 217172, upload-time = "2026-07-29T22:46:04.895Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/06/79/b4c714bef36bc4ec2beeae1e0c124f0223888cd8c6feb1cdc56038116920/filelock-3.32.0-py3-none-any.whl", hash = "sha256:d396bea984af47333ef05e50eae7eff88c84256de6112aea0ec48a233c064fe3", size = 97732, upload-time = "2026-07-21T13:17:41.55Z" }, + { url = "https://files.pythonhosted.org/packages/c1/e8/72f8cef9fdfeffe06213fe8508039396ee48daa0e3259457ed766173bfd6/filelock-3.32.2-py3-none-any.whl", hash = "sha256:87dd94cf281e586d135fa51132b8e3d9a598b316e90377a288663c9321036c82", size = 98830, upload-time = "2026-07-29T22:46:03.52Z" }, ] [[package]] @@ -887,7 +887,7 @@ name = "importlib-metadata" version = "9.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "zipp", marker = "python_full_version < '3.13'" }, + { name = "zipp" }, ] sdist = { url = "https://files.pythonhosted.org/packages/a9/01/15bb152d77b21318514a96f43af312635eb2500c96b55398d020c93d86ea/importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc", size = 56405, upload-time = "2026-03-20T06:42:56.999Z" } wheels = [ @@ -1249,26 +1249,26 @@ wheels = [ [[package]] name = "prek" -version = "0.4.11" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/c4/1a/73b6dae5ce7e997cb35a69bfe1d25a798e85fa3d2eabf95324563f461b30/prek-0.4.11.tar.gz", hash = "sha256:4a14cb9bbae850605ae3904fbdbb12f0e00c12455efaa2266da8fb8e5c0350d7", size = 516254, upload-time = "2026-07-24T17:05:35.107Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/8e/d7/a00b2de492a80e99b1698e72c1d196ac3ed544dc7ee0ada261ac066e78e0/prek-0.4.11-py3-none-linux_armv6l.whl", hash = "sha256:3830cb7cc47e837888b8b464ecb21355a69235cfacef0fd89101e17f09345d63", size = 5770511, upload-time = "2026-07-24T17:05:12.829Z" }, - { url = "https://files.pythonhosted.org/packages/c1/1e/f97c74defcd5d5645888cb99fcdd9b8b48cc7247cf31e64414d106d56d66/prek-0.4.11-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:45facadf9c2332b28e6ab2744312ae0275a93ab5a437da8bcc53a8c7260cb4b0", size = 6118049, upload-time = "2026-07-24T17:05:14.451Z" }, - { url = "https://files.pythonhosted.org/packages/d9/92/8367d26421ee6fe6019a63928fb0ed31179cd0d6199879c524f89ef4c95c/prek-0.4.11-py3-none-macosx_11_0_arm64.whl", hash = "sha256:2f8a194b1d00d24dff8baff691c99e2668339da2da3482b4ee99c1a0f2409378", size = 5601478, upload-time = "2026-07-24T17:05:15.81Z" }, - { url = "https://files.pythonhosted.org/packages/e1/6f/7617c9b87afaadede4167720aae7ef12d7e51167db903bc8fbac0cadef7b/prek-0.4.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:21d93e6d76bf3d7a9bb70c6ac86ef372adaee50068c45749e6b9e1c2ac4ec939", size = 5932071, upload-time = "2026-07-24T17:05:17.217Z" }, - { url = "https://files.pythonhosted.org/packages/ef/41/6796a4011b04212333259064aa885a71d03d9581ba9bff52db3ce58d1f06/prek-0.4.11-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7fb07cde2d2156efa6980122b3f13dc88f20c84b933c6205675d0f1e7be2cde8", size = 5677617, upload-time = "2026-07-24T17:05:18.658Z" }, - { url = "https://files.pythonhosted.org/packages/03/5f/3e339901f8460b6073313619b4fd9bf4135e48430695c53640b83ace88ea/prek-0.4.11-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6b7f5e446d2aca739bd18b380578e9c78bb4c086299abc3e167d51c47840c56f", size = 6106370, upload-time = "2026-07-24T17:05:20.219Z" }, - { url = "https://files.pythonhosted.org/packages/8e/4e/94e24b5c1910ec15692ecaf33d0d8ef0d02a02a8b5e40d3c976396880cba/prek-0.4.11-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b7059d640e595d098600e2d97af961f46292b4112670edbe632de84f6828389e", size = 6884342, upload-time = "2026-07-24T17:05:21.587Z" }, - { url = "https://files.pythonhosted.org/packages/a2/7c/fc0daa033dcafe74990c00af2da1e16922790b9c1b8da7e8eebf1123838b/prek-0.4.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:85a4df33998fcac878bce3b2c624e1caeb9609f3d562c640702c1234ed815daf", size = 6331365, upload-time = "2026-07-24T17:05:22.87Z" }, - { url = "https://files.pythonhosted.org/packages/d7/36/49f152b8f539930e9685cff0509695ce4054abcecc22f4c16c4e1e5c23d0/prek-0.4.11-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:866991f387527c5f880ce2cc3ddea9b33cc5b986881f8c7f92524cf0969c1350", size = 5939075, upload-time = "2026-07-24T17:05:24.249Z" }, - { url = "https://files.pythonhosted.org/packages/4f/fe/7b097af9161edae7ddedcc9f5cda4a5f31346a492ce3f92583d96c46f628/prek-0.4.11-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:247e5d8740e137ebdf24fa96f01a95b2d2f1892e956a1ab00c7b1474a59ebcc0", size = 5799029, upload-time = "2026-07-24T17:05:25.652Z" }, - { url = "https://files.pythonhosted.org/packages/49/63/dc955ff99e1002d3cd375b21467c87046bc41deddfce86d898240757b151/prek-0.4.11-py3-none-musllinux_1_1_armv7l.whl", hash = "sha256:603ba9f2fd9d666dddb3ae190a25a5c55091b843cde90ed52e0a1116f50d4062", size = 5651211, upload-time = "2026-07-24T17:05:27.027Z" }, - { url = "https://files.pythonhosted.org/packages/8f/b8/bf2139ec25eefb5afef43afac7620c5181f2c2661f220598beacb1771207/prek-0.4.11-py3-none-musllinux_1_1_i686.whl", hash = "sha256:f2682ece3c5fc7201106c4fdd84b0587ccea4b8a2ecefa3e94d3074d2841f1df", size = 5954784, upload-time = "2026-07-24T17:05:28.391Z" }, - { url = "https://files.pythonhosted.org/packages/fc/29/3fe5990aee1bd7c4d50a03358ad867c5e912d9510aafb83a359c7347e5fa/prek-0.4.11-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:22721d30394192931fecc80d6fc6dd47e8ddf8db8b9693805aba8ec0f50087ec", size = 6448916, upload-time = "2026-07-24T17:05:29.837Z" }, - { url = "https://files.pythonhosted.org/packages/0f/fb/abddacf43738302242ecd237236c7c80cd7c1d27545e16e803770aee76e9/prek-0.4.11-py3-none-win32.whl", hash = "sha256:8b093e7624522146049e994d5cf283d01d71632b638620b79ae0f96afeaa2624", size = 5483539, upload-time = "2026-07-24T17:05:31.228Z" }, - { url = "https://files.pythonhosted.org/packages/00/1e/c293f7a15cb93963c4be36a02e144b93f43906bc02644a3f04e5708e7453/prek-0.4.11-py3-none-win_amd64.whl", hash = "sha256:5a3d7c80b970b456e5f1bcec8382008ee1ae6a3f324a6b9bb4ff7e666ab0f3c4", size = 5861119, upload-time = "2026-07-24T17:05:32.479Z" }, - { url = "https://files.pythonhosted.org/packages/cd/0c/05fe6eb9d6a54d0e02dfa8cc5ad6f86869bf953419ec15909a32466a28ee/prek-0.4.11-py3-none-win_arm64.whl", hash = "sha256:e7b0df37ce05e45a14a9da39ab104691474d72f139bf4f6c860f754763a322cb", size = 5626386, upload-time = "2026-07-24T17:05:33.813Z" }, +version = "0.4.12" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/fc/5c/cb6e63f7e5a58a5313ddb70409174f4dc004e4b0910b8a8d3f59b2225a95/prek-0.4.12.tar.gz", hash = "sha256:04beeba7f40437cd2f36804b84101bd7f3c9fb40b52da46a25604642ab2bfb09", size = 519080, upload-time = "2026-08-03T11:28:33.147Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/23/5811a3161e072e5f93e4da01af611ee30c32922507b8ab4d9873df6affd3/prek-0.4.12-py3-none-linux_armv6l.whl", hash = "sha256:cd92000b051e433f26340821cf1cc8e6e3960f1275f3d516ca01f05905abba64", size = 5793226, upload-time = "2026-08-03T11:28:09.534Z" }, + { url = "https://files.pythonhosted.org/packages/a3/88/8607845d94eb1482e1bd335dadf098618f077a15775f7e98de99669052b4/prek-0.4.12-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:5904fe6c6ab26e7d8792a3c7f1e3fc8d94fcfb63ad33b247c35f004b62cb6275", size = 6132269, upload-time = "2026-08-03T11:28:11.147Z" }, + { url = "https://files.pythonhosted.org/packages/ac/28/571d79ba457fbd9ecf40ae879c91952e12f5fa475306218c91139b86db7a/prek-0.4.12-py3-none-macosx_11_0_arm64.whl", hash = "sha256:df3eff1db9c24dc293010a07bc7a0ae0c541d55af828f5586405dedc28c4920d", size = 5614964, upload-time = "2026-08-03T11:28:12.983Z" }, + { url = "https://files.pythonhosted.org/packages/b0/a9/3f5cb79a73c764a8ac38d5bcd51e0df57239856eca7949b09bdac4338bf3/prek-0.4.12-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:c7733b44ca772ea32ec6a8bee669d0358bdf45873e79767afed196065084f31c", size = 5941047, upload-time = "2026-08-03T11:28:14.45Z" }, + { url = "https://files.pythonhosted.org/packages/8c/00/1dfed0ef8af10c5c32aa903486dccd33d2df171f3d945a037c5692f10760/prek-0.4.12-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:87f170cf1ffd6e3a196f947b83dff1f6c2cd68635f8d49740278bebe7b682262", size = 5707994, upload-time = "2026-08-03T11:28:15.914Z" }, + { url = "https://files.pythonhosted.org/packages/c0/bd/5f388f6cbdc0445b850e7c1a160d0be67fcef8bf221e3c8141a1feccef17/prek-0.4.12-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:57dad513831f060cf73808df8edec29d46ec311435aa69f21c80edebf23dc5e1", size = 6133784, upload-time = "2026-08-03T11:28:17.184Z" }, + { url = "https://files.pythonhosted.org/packages/ba/47/342091a987bf68a74acec6d226a40ce7d51faf0019aa4126cc7bc952f8a7/prek-0.4.12-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b204844abc7ded983471f576ae8dc13b99e9b8d022e4d4b46176c6654769c9d8", size = 6901589, upload-time = "2026-08-03T11:28:18.545Z" }, + { url = "https://files.pythonhosted.org/packages/5b/8a/3ef7bdc3c3441649ebc040b9e164a13163e1e5fabae23e7bbb901992f3de/prek-0.4.12-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:43b0a5a9d3f2f77871fdcb7893bfc5c8fe7e44f4e603ce6e4712bfec96b2d6f2", size = 6342189, upload-time = "2026-08-03T11:28:20Z" }, + { url = "https://files.pythonhosted.org/packages/c4/da/6277908442301b1b92a2879f6b04aaa03accb900f80e42776fc28b8197ef/prek-0.4.12-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:0d188e572c306cc44b96e1bae5647e25b7bd311113f3f3f4a67320c257ee64a3", size = 5951250, upload-time = "2026-08-03T11:28:21.339Z" }, + { url = "https://files.pythonhosted.org/packages/a3/68/bff51a7332837edb1ecbe017325adb7fafd69b9c7828ddc81a1334b884af/prek-0.4.12-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:986f52d104b7066190f0f32aebe3467710356de265e9bfd892101ba99371db4d", size = 5804147, upload-time = "2026-08-03T11:28:22.656Z" }, + { url = "https://files.pythonhosted.org/packages/aa/de/b7f544971072ed7814125145dfeb1f7c15cce6b78ccea65a96298ff37838/prek-0.4.12-py3-none-musllinux_1_1_armv7l.whl", hash = "sha256:13e34d9e09bafcbf1f25a01cf86985e2c5e486591d3f45b2786ba3de82e5153a", size = 5680104, upload-time = "2026-08-03T11:28:24.271Z" }, + { url = "https://files.pythonhosted.org/packages/68/94/95942bcc20a6a91ec2989aa30fdeb00ad095be736ec48b4bbcf0376166b1/prek-0.4.12-py3-none-musllinux_1_1_i686.whl", hash = "sha256:3d0208370da73e8b5bc97f2492dc3975f8dd2c22f4bf6e1f2cf3342503764b52", size = 5975030, upload-time = "2026-08-03T11:28:25.683Z" }, + { url = "https://files.pythonhosted.org/packages/ef/6d/26e6497198d81cf9aa82495400aef46adea8df3e4a4efc5f00e3b6ab3292/prek-0.4.12-py3-none-musllinux_1_1_x86_64.whl", hash = "sha256:b1005f42920111bec1403c25e8f2f12ec7af0be06686cc3b8dcf85429af908a8", size = 6458532, upload-time = "2026-08-03T11:28:27.121Z" }, + { url = "https://files.pythonhosted.org/packages/44/02/ee140c2eb4701bd194db429d84630733492be94897d5f72b61d6f11e6619/prek-0.4.12-py3-none-win32.whl", hash = "sha256:afee229488dcceaea282288e4d7096a93da5a8b85649d9ef506dbdbcd78f38a7", size = 5502213, upload-time = "2026-08-03T11:28:28.691Z" }, + { url = "https://files.pythonhosted.org/packages/e5/7b/744cff84def48c1ce38c0b4f643a3553c66976c5bb7869ab7317044870e4/prek-0.4.12-py3-none-win_amd64.whl", hash = "sha256:fdd27bad8adafea8fe77606950ca09200d59296a47ab131cfb88718d460949d7", size = 5868065, upload-time = "2026-08-03T11:28:30.377Z" }, + { url = "https://files.pythonhosted.org/packages/46/1d/e2c0fc222904ef73df1739b11a83edc29e38bc4bc61259f2ca6d2f15abb0/prek-0.4.12-py3-none-win_arm64.whl", hash = "sha256:45e34a24fba4a4e4568682477158591698efc2375b8d1d418ae424691c4bd01b", size = 5632819, upload-time = "2026-08-03T11:28:31.743Z" }, ] [[package]] diff --git a/dev/react-plugin-tools/react_plugin_template/package.json b/dev/react-plugin-tools/react_plugin_template/package.json index f2b6a495b7daa..d6a9fde434a08 100644 --- a/dev/react-plugin-tools/react_plugin_template/package.json +++ b/dev/react-plugin-tools/react_plugin_template/package.json @@ -46,12 +46,12 @@ "@testing-library/react": "^16.3.0", "@trivago/prettier-plugin-sort-imports": "^6.0.2", "@types/node": "^25.9.5", - "@types/react": "^19.2.17", - "@types/react-dom": "^19.2.3", + "@types/react": "^19.2.18", + "@types/react-dom": "^19.2.4", "@typescript-eslint/eslint-plugin": "8.65.0", "@typescript-eslint/parser": "8.65.0", "@typescript-eslint/utils": "^8.65.0", - "@vitejs/plugin-react-swc": "^4.3.2", + "@vitejs/plugin-react-swc": "^4.3.3", "@vitest/coverage-v8": "^3.2.7", "eslint": "^9.39.5", "eslint-config-prettier": "^10.1.2", @@ -62,12 +62,12 @@ "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.5.3", "eslint-plugin-unicorn": "^55.0.0", - "globals": "^17.8.0", + "globals": "^17.9.0", "happy-dom": "^20.11.1", "prettier": "^3.9.6", "typescript": "~5.9.3", "typescript-eslint": "^8.65.0", - "vite": "^8.1.5", + "vite": "^8.2.0", "vite-plugin-css-injected-by-js": "^4.0.1", "vite-plugin-dts": "^4.3.0", "vitest": "^4.1.10" diff --git a/dev/react-plugin-tools/react_plugin_template/pnpm-lock.yaml b/dev/react-plugin-tools/react_plugin_template/pnpm-lock.yaml index dbf3d2f0d0af3..b3c1baf5cb61f 100644 --- a/dev/react-plugin-tools/react_plugin_template/pnpm-lock.yaml +++ b/dev/react-plugin-tools/react_plugin_template/pnpm-lock.yaml @@ -37,10 +37,10 @@ importers: version: 2.3.4 '@chakra-ui/react': specifier: ^3.36.1 - version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@emotion/react': specifier: ^11.14.0 - version: 11.14.0(@types/react@19.2.17)(react@19.2.8) + version: 11.14.0(@types/react@19.2.18)(react@19.2.8) next-themes: specifier: ^0.4.6 version: 0.4.6(react-dom@19.2.8(react@19.2.8))(react@19.2.8) @@ -65,7 +65,7 @@ importers: version: 6.9.1 '@testing-library/react': specifier: ^16.3.0 - version: 16.3.2(@testing-library/dom@10.4.1)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) + version: 16.3.2(@testing-library/dom@10.4.1)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@trivago/prettier-plugin-sort-imports': specifier: ^6.0.2 version: 6.0.2(prettier@3.9.6) @@ -73,11 +73,11 @@ importers: specifier: ^25.9.5 version: 25.9.5 '@types/react': - specifier: ^19.2.17 - version: 19.2.17 + specifier: ^19.2.18 + version: 19.2.18 '@types/react-dom': - specifier: ^19.2.3 - version: 19.2.3(@types/react@19.2.17) + specifier: ^19.2.4 + version: 19.2.4(@types/react@19.2.18) '@typescript-eslint/eslint-plugin': specifier: 8.65.0 version: 8.65.0(@typescript-eslint/parser@8.65.0(eslint@9.39.5)(typescript@5.9.3))(eslint@9.39.5)(typescript@5.9.3) @@ -88,8 +88,8 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@9.39.5)(typescript@5.9.3) '@vitejs/plugin-react-swc': - specifier: ^4.3.2 - version: 4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + specifier: ^4.3.3 + version: 4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) '@vitest/coverage-v8': specifier: ^3.2.7 version: 3.2.7(vitest@4.1.10) @@ -121,8 +121,8 @@ importers: specifier: ^55.0.0 version: 55.0.0(eslint@9.39.5) globals: - specifier: ^17.8.0 - version: 17.8.0 + specifier: ^17.9.0 + version: 17.9.0 happy-dom: specifier: ^20.11.1 version: 20.11.1 @@ -136,17 +136,17 @@ importers: specifier: ^8.65.0 version: 8.65.0(eslint@9.39.5)(typescript@5.9.3) vite: - specifier: ^8.1.5 - version: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + specifier: ^8.2.0 + version: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) vite-plugin-css-injected-by-js: specifier: ^4.0.1 - version: 4.0.1(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + version: 4.0.1(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) vite-plugin-dts: specifier: ^4.3.0 - version: 4.5.4(@types/node@25.9.5)(rollup@4.59.0)(typescript@5.9.3)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + version: 4.5.4(@types/node@25.9.5)(rollup@4.59.0)(typescript@5.9.3)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) vitest: specifier: ^4.1.10 - version: 4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + version: 4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) packages: @@ -247,15 +247,6 @@ packages: react: '>=18' react-dom: '>=18' - '@emnapi/core@1.11.1': - resolution: {integrity: sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==} - - '@emnapi/runtime@1.11.1': - resolution: {integrity: sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==} - - '@emnapi/wasi-threads@1.2.2': - resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} - '@emotion/babel-plugin@11.13.5': resolution: {integrity: sha512-pxHCpT2ex+0q+HH91/zsdHkw/lXd468DIN2zvfvLtPKLLMo6gQj7oLObq8PhkrxOZb/gGCq03S3Z7PDhS8pduQ==} @@ -422,15 +413,8 @@ packages: '@microsoft/tsdoc@0.16.0': resolution: {integrity: sha512-xgAyonlVVS+q7Vc7qLW0UrJU7rSFcETRWsqdXZtjzRU8dF+6CkozTK4V4y1LwOX7j8r/vHphjDeMeGI4tNGeGA==} - '@napi-rs/wasm-runtime@1.2.2': - resolution: {integrity: sha512-JfB4kuJQjaoHuCTseIINHtHWeJnvgEcxjwA5t/Y00ZgaOO1Crz3fjT/p8kT28zA/Caz7oiUMn3d6H2yOVCVwuw==} - engines: {node: ^20.19.0 || ^22.13.0 || >=23.5.0} - peerDependencies: - '@emnapi/core': ^1.7.1 || ^2.0.0-alpha.3 - '@emnapi/runtime': ^1.7.1 || ^2.0.0-alpha.3 - - '@oxc-project/types@0.139.0': - resolution: {integrity: sha512-r9gHphtCs+1M7J0pw6Sn/hh/Wpa/iQrOOkrNAlVLF/gHq+/CJmHIWKKUUhdWjcD6CIa8idarspCsASiXCXvFUw==} + '@oxc-project/types@0.143.0': + resolution: {integrity: sha512-u6JZdLBTLotrNC9Vd6vPssINdzcCzleKAH6EJKImQb7GtYvX5keN2dxkoK44stCc4tffE6QQRtZTXVSzsLUlWA==} '@pandacss/is-valid-prop@1.12.0': resolution: {integrity: sha512-P7tXdSwpJz9sTqQTtuC6V7lyaEGEc1M42JxRbn5MSdpIzjQzIXOS45aVBDYQuX7ADiMjGvkwRP1bJsnJZGPnkQ==} @@ -444,97 +428,92 @@ packages: resolution: {integrity: sha512-SEeaJLb3qBNF/OaXnaR1NmmBbFYk1zC0ZH/52fATcRPLFg/p791YrcyFFy44Bo9sLaGuSuLp5Q6axbb/O+v/RA==} engines: {node: ^14.18.0 || >=16.0.0} - '@rolldown/binding-android-arm64@1.1.5': - resolution: {integrity: sha512-lZg8fqIv2v7FF237bwMgzGZEJvGL79/s5knJ/i6FmsGF4XXlzccZ4jb+TrFIxtSSxFtIpdsgrPZeMk1I9AFcyQ==} + '@rolldown/binding-android-arm64@1.2.3': + resolution: {integrity: sha512-zrJtHDcaZJ1Fp7xf4hNl+7seH9Cn/N5TwLYkhgXREtBwAd/jaqW3uqeHxpDugJLVICWg4eW44kOQEGJ1r6jCGw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [android] - '@rolldown/binding-darwin-arm64@1.1.5': - resolution: {integrity: sha512-51Bnx9pNiMRKSUNtBfySkNJ9vMU9Hh3I1ozDd6gyPPYzaXCfnptUcEZxXGYFn+ul2dtcMUiqGR1Yai2K10uoTw==} + '@rolldown/binding-darwin-arm64@1.2.3': + resolution: {integrity: sha512-ieIiibVCp0tX7TLu2cafoNPv8wJyYi01ekXpbf8q2j7F4rGAhhXb/eQh7ge9DRBY78GwmRQtvjZDux7EDbA8kA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [darwin] - '@rolldown/binding-darwin-x64@1.1.5': - resolution: {integrity: sha512-Tm+gbfC0aHu1tBA/JvKQh32S0K6YgCHkiAF4/W6xX0K0RmNuc94VeK419dJoE65R5aRxmo+noZQSWrAMF6yb6g==} + '@rolldown/binding-darwin-x64@1.2.3': + resolution: {integrity: sha512-Zh9tCon19eDXJoihx0rqKhMUlMYqzwj3aPsSuHmI4RWZh62dWUL+DJN4C5YQya5TcQBJU/Fe8+rY0jhXTQITqA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [darwin] - '@rolldown/binding-freebsd-x64@1.1.5': - resolution: {integrity: sha512-JMzDKCCXq93YccG5gz3hvOs1oXRKAf0XYpfOS88e+wZrC8Iugj6j68867vrYZkvpDDpKn/KoKORThmchMpF6TA==} + '@rolldown/binding-freebsd-x64@1.2.3': + resolution: {integrity: sha512-nGbJWewA1wrXXZiQhjAT5rhibGfns5ZNkDVqxsO6zJ3f3YvpoDNNmGMSbbhLuXKjNScaBJVOAboztAWVespQMg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [freebsd] - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': - resolution: {integrity: sha512-uML21j2K5TfPGutKxub+M+nLjZIrWjXQ5Grx4lCe/nimTj9B4L63zHpjXLl4y0L3mcm2htEQIb06oCG/szerNw==} + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': + resolution: {integrity: sha512-QNniJr5Kml0kDEB98jiDOJjXNroxIIi0IXIbdYzY26Xt1pVbeP62+KnoIZLwirOymX/0jDk/2gI/bNUv7A7OIw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm] os: [linux] - '@rolldown/binding-linux-arm64-gnu@1.1.5': - resolution: {integrity: sha512-navSiuTMogvnQoZoM/v+l3ZWo50/NTwSHSzheABx/RCnmUPaKwq9qSo4Br2OYRs21+Fz8uFqITZM3H4opOB0/Q==} + '@rolldown/binding-linux-arm64-gnu@1.2.3': + resolution: {integrity: sha512-TkqEAcmmvH3I/q4114NB4RVt6241Dao48pF45uLcFGrwAaIn0iITgTAKP/dLjbN0R4buJjGb91+UHSoFmpgIWw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-arm64-musl@1.1.5': - resolution: {integrity: sha512-lAryqH7IteztmCXQXk0etKj4wBQ7Gx5S6LjKhsgp9zb8I5bsuvU/2llH1hDQcjsFeqIsovMVN339/8pUDDBXxA==} + '@rolldown/binding-linux-arm64-musl@1.2.3': + resolution: {integrity: sha512-NHqjnxpsndf4MPymxteFAWHHfkTL8HjWh1KB7z23ofZ6QO2euONuxDXjat69dKZRALnGypg8k8SsK8vZJoXv1Q==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] libc: [musl] - '@rolldown/binding-linux-ppc64-gnu@1.1.5': - resolution: {integrity: sha512-fsK/sNBnxzBlL4O1JNrZakVQxPspqpED5dLtNsZS9oOKmtSpdNIzxH2kkol5HYTWJN47sE20ztMJPxfZ89qGOg==} + '@rolldown/binding-linux-ppc64-gnu@1.2.3': + resolution: {integrity: sha512-6tbrbwfz5GB9DQ4Jwo6hy9v+vR31xZlvzZ6n5Xut6Hhx5PvrA9q/HsK8KMaYQp063iqZGXwNvZtYNLD7EM/x0w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-s390x-gnu@1.1.5': - resolution: {integrity: sha512-gLYb4BIadlfTOYT5gO503n8zQjXflgzpD0FcyKh0Mzx3rqCZKnHoJWV9xe1KXUJ5lx2JfcSHr/mhzS0PC/McAA==} + '@rolldown/binding-linux-s390x-gnu@1.2.3': + resolution: {integrity: sha512-oyuXxXmoZHjXC917IAPFAAv4wWAa0cM9afk8nx1+9/jNNOX1uPf8yDA6p7G0RypOfw/X0PQt5IfoquY1um+zSg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] libc: [glibc] - '@rolldown/binding-linux-x64-gnu@1.1.5': - resolution: {integrity: sha512-FjcpEKUyJygHgs1o50VYNvkt5+7Le/VEdYt0AkRpkL33MnyQfwr8l5mXwMmfmTbyMPr5vJLC+8/Gd9gXnwU1QQ==} + '@rolldown/binding-linux-x64-gnu@1.2.3': + resolution: {integrity: sha512-TytMwF2KVGqP2tgd0I1OY0PAv78dZRAYcF5ssDzjM34SUXCED3uXvSd5+lHoC0bTD6eEdFz7LdQNCO1y0oVk9w==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [glibc] - '@rolldown/binding-linux-x64-musl@1.1.5': - resolution: {integrity: sha512-Me+PfPI2TMeOQk0gYWfLQZtTktrmzbr8cDboqX83XKc7UrgAi55gF+2dUkWdxd19n55Essp2yeca+O9N5rBxHg==} + '@rolldown/binding-linux-x64-musl@1.2.3': + resolution: {integrity: sha512-/E9m3qstrJFVPoULV25mVQblSNExY2+kBsYe4sy0Tn0yOOgJ8wZbZt3KnRbF/XeU2Gl1STKUQnDNTqhIE5MD4A==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] libc: [musl] - '@rolldown/binding-openharmony-arm64@1.1.5': - resolution: {integrity: sha512-yc5WrLzXks6zCQfn9Oxr8pORKyl/pF+QjHmW/Qx3qu0oyrrNC+y2JLTU1E2rcWYAmzlnqngWXHQjy51VzW70Vw==} + '@rolldown/binding-openharmony-arm64@1.2.3': + resolution: {integrity: sha512-Kr0OcsoQI816i6HOl3vFHpd1K0eZyh76zgfj4c1nTyaTsd5r2Mj1lwM4R90y/qaCfmTn9eHy0SKwi98eitRxug==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [openharmony] - '@rolldown/binding-wasm32-wasi@1.1.5': - resolution: {integrity: sha512-VbQGPX2b4r48TAMIM2cjgluIM1HYutm4pcTEJsle7iEP7sB1dFqtPLBVbdLAZCxy1txCcPxf4QFf4v8uvltPqA==} - engines: {node: ^20.19.0 || >=22.12.0} - cpu: [wasm32] - - '@rolldown/binding-win32-arm64-msvc@1.1.5': - resolution: {integrity: sha512-gHv82k63z4qpV5+Q1y/12KrK0ltWBukVDI8nZcbT7Tt/ZlOIVwppazneq0F93oDxTo3IgAMEDIoQh3E2n6mVsw==} + '@rolldown/binding-win32-arm64-msvc@1.2.3': + resolution: {integrity: sha512-hOtMwTqnME+/gJcH/PCZ0wn0zPUjiWOgkHpxbSJpfGKMezHltx1S7/k1SitzVa7Ww2cqrDDaFbZEhcJZO8o+Jw==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [win32] - '@rolldown/binding-win32-x64-msvc@1.1.5': - resolution: {integrity: sha512-tTZuDBPw85tEN5PQi1pnEBzDy0Z49HtScLAbD5t6hyeU92A95pRWaSMw1GZZi/RwgSgUIl0xrSlXIT/9QzvYSA==} + '@rolldown/binding-win32-x64-msvc@1.2.3': + resolution: {integrity: sha512-ekcqMMkI2PlhYnfzQnB/cEdYUVVJViWvoUyLrbzgDoi3Snfc1mVBwdnc306ufA5ejy8JSPjT2RlW1nQSjW7efg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [win32] @@ -866,9 +845,6 @@ packages: svelte: optional: true - '@tybys/wasm-util@0.10.3': - resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} - '@types/argparse@1.0.38': resolution: {integrity: sha512-ebDJ9b0e702Yr7pWgB0jzm+CX4Srzz8RcXtLJDJB+BSccqMa36uyH/zUsSYao5+BD1ytv3k3rPYCq4mAE1hsXA==} @@ -899,13 +875,13 @@ packages: '@types/parse-json@4.0.2': resolution: {integrity: sha512-dISoDXWWQwUquiKsyZ4Ng+HX2KsPL7LyHKHQwgGFEA3IaKac4Obd+h2a/a6waisAoepJlBcx9paWqjA8/HVjCw==} - '@types/react-dom@19.2.3': - resolution: {integrity: sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==} + '@types/react-dom@19.2.4': + resolution: {integrity: sha512-Bsc+QHgp+P/F02XDzNCY9jnZNCUuLki36KT7VKrTXXLdHf+vHMNZnW1rVu5DNW/rCK+fya3DATySbLM4yhtKUw==} peerDependencies: '@types/react': ^19.2.0 - '@types/react@19.2.17': - resolution: {integrity: sha512-MXfmqaVPEVgkBT/aY0aGCkRWWtByiYQXo3xdQ8r5RzuFrPiRn8Gar2tQdXSUQ2GKV3bkXckek89V8wQBY2Q/Aw==} + '@types/react@19.2.18': + resolution: {integrity: sha512-AnzbBERsrLKtk2XSfTbYRLjQPdy116Sty4q+T+Bp3IC4l6jNBvreVPAHmpq9qhXQM7CXZPjLVmGMw9sy+hxQ3w==} '@types/whatwg-mimetype@3.0.2': resolution: {integrity: sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA==} @@ -972,8 +948,8 @@ packages: resolution: {integrity: sha512-8C71BQkGjiMmXtop7pHVJu1l2NNShFdkCyD6a2ezzs5vU/L3LRtb69EtcteFwz0mYMPzIgOw0n6OV4VBUWZd7A==} engines: {node: ^18.18.0 || ^20.9.0 || >=21.1.0} - '@vitejs/plugin-react-swc@4.3.2': - resolution: {integrity: sha512-MlSlmSAYbYwPjGa+CjOAqwij09iPYAQDHwx8osVkwoDamCxXlg+avpkC65Ysv+L/uqUWvRTsnJCKPjVhZih/sA==} + '@vitejs/plugin-react-swc@4.3.3': + resolution: {integrity: sha512-bti8ZAcvz4Lh6/e4Uk2k3aa1TiUXbbMsahuqOHvd3MveFTkKDZOA6wQVkpj7J/+tepX/wGfe+lsGh/t24HTXMQ==} engines: {node: ^20.19.0 || >=22.12.0} peerDependencies: vite: '>=7.3.2' @@ -1859,8 +1835,8 @@ packages: resolution: {integrity: sha512-7ACyT3wmyp3I61S4fG682L0VA2RGD9otkqGJIwNUMF1SWUombIIk+af1unuDYgMm082aHYwD+mzJvv9Iu8dsgg==} engines: {node: '>=18'} - globals@17.8.0: - resolution: {integrity: sha512-Zz/LMDZScFmkakeL2cTHzf+PbWKdpU3uclqkZT7TjDG58j5WPt0PpA+n9uPI24fZtlw07q0OtEi84K+umsRzqQ==} + globals@17.9.0: + resolution: {integrity: sha512-m/MvAW61QVU5VDNF1Vj8axt016h8w7L5TU1e9zlab7XIttAT2YAlCwl75K1fOqvMM9apmD7lbCIRhpfkhmxhCg==} engines: {node: '>=18'} globalthis@1.0.4: @@ -2298,8 +2274,8 @@ packages: muggle-string@0.4.1: resolution: {integrity: sha512-VNTrAak/KhO2i8dqqnqnAHOa3cYBwXEZe9h+D5h/1ZqFSTEFHdM65lR7RoIqq3tBBYavsOXV84NoHXZ0AkPyqQ==} - nanoid@3.3.16: - resolution: {integrity: sha512-bzlKTyNJ7+LdGIIwy8ijFpIqEQIvafahV7eYykJ8Cvh42EdJeODoJ6gUJXpQJvej1BddH8OqTXZNE/KfbWAu8Q==} + nanoid@3.3.17: + resolution: {integrity: sha512-xQLf0A3HOMlgHq0n247/LRuAOYmB7dXJ/DvAxGvsSBij45XtBSmQycu+F8ODbHwns/XyFZagyL1+J0Offw1E0g==} engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true @@ -2456,8 +2432,8 @@ packages: resolution: {integrity: sha512-/+5VFTchJDoVj3bhoqi6UeymcD00DAwb1nJwamzPvHEszJ4FpF6SNNbUbOS8yI56qHzdV8eK0qEfOSiodkTdxg==} engines: {node: '>= 0.4'} - postcss@8.5.25: - resolution: {integrity: sha512-DTPx3RWSSnWyzLxQnlH0rJP+EW5ekl16ZU4/psbIhA0e53kJfdgaN5vKM+xP7yJtXVu+nfdVFmlgFDEKAe4Pyw==} + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} engines: {node: ^10 || ^12 || >=14} prelude-ls@1.2.1: @@ -2554,8 +2530,8 @@ packages: engines: {node: '>= 0.4'} hasBin: true - rolldown@1.1.5: - resolution: {integrity: sha512-t9z29cJjXf/vxQ8dyhCSpt6H6aSwHTk8cT5I3iy6SMXuFpk5mB6PL6XfC8PCwrPTx93udwKUm9HRteAlTGBLiA==} + rolldown@1.2.3: + resolution: {integrity: sha512-rn9wpmxplLf7NLNyCk9FyWh3FM43DbY8jOzCdEPzH7uflhTftRbCEpqi6Ly2osgoU8OwObtmavMbWLaWy4LX7A==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true @@ -2875,13 +2851,13 @@ packages: vite: optional: true - vite@8.1.5: - resolution: {integrity: sha512-7ULLwsCdYx/nRyrpiEwvqb5TFHrMVZyBt+rg/OAXT7rgj/z+DtTDyKFeLAdDkubDVDKD8jOsndmy7m55XcfUsw==} + vite@8.2.0: + resolution: {integrity: sha512-pn+CFpM0lwDeKwmOq1ZaBK/9sjorZcgqxki6MbY/jPEVd9vichIlmlD4HmQ5wdP5EgqQCFRaACBxMC7uEGc6lQ==} engines: {node: ^20.19.0 || >=22.12.0} hasBin: true peerDependencies: '@types/node': ^20.19.0 || >=22.12.0 - '@vitejs/devtools': ^0.3.0 + '@vitejs/devtools': ^0.4.0 esbuild: '>=0.28.1' jiti: '>=1.21.0' less: ^4.0.0 @@ -3190,11 +3166,11 @@ snapshots: '@chakra-ui/anatomy@2.3.4': {} - '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@chakra-ui/react@3.36.1(@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8))(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@ark-ui/react': 5.37.2(react-dom@19.2.8(react@19.2.8))(react@19.2.8) '@emotion/is-prop-valid': 1.4.0 - '@emotion/react': 11.14.0(@types/react@19.2.17)(react@19.2.8) + '@emotion/react': 11.14.0(@types/react@19.2.18)(react@19.2.8) '@emotion/serialize': 1.3.3 '@emotion/use-insertion-effect-with-fallbacks': 1.2.0(react@19.2.8) '@emotion/utils': 1.4.2 @@ -3203,22 +3179,6 @@ snapshots: react: 19.2.8 react-dom: 19.2.8(react@19.2.8) - '@emnapi/core@1.11.1': - dependencies: - '@emnapi/wasi-threads': 1.2.2 - tslib: 2.8.1 - optional: true - - '@emnapi/runtime@1.11.1': - dependencies: - tslib: 2.8.1 - optional: true - - '@emnapi/wasi-threads@1.2.2': - dependencies: - tslib: 2.8.1 - optional: true - '@emotion/babel-plugin@11.13.5': dependencies: '@babel/helper-module-imports': 7.28.6 @@ -3251,7 +3211,7 @@ snapshots: '@emotion/memoize@0.9.0': {} - '@emotion/react@11.14.0(@types/react@19.2.17)(react@19.2.8)': + '@emotion/react@11.14.0(@types/react@19.2.18)(react@19.2.8)': dependencies: '@babel/runtime': 7.28.6 '@emotion/babel-plugin': 11.13.5 @@ -3263,7 +3223,7 @@ snapshots: hoist-non-react-statics: 3.3.2 react: 19.2.8 optionalDependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 transitivePeerDependencies: - supports-color @@ -3440,14 +3400,7 @@ snapshots: '@microsoft/tsdoc@0.16.0': {} - '@napi-rs/wasm-runtime@1.2.2(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1)': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@tybys/wasm-util': 0.10.3 - optional: true - - '@oxc-project/types@0.139.0': {} + '@oxc-project/types@0.143.0': {} '@pandacss/is-valid-prop@1.12.0': {} @@ -3456,53 +3409,46 @@ snapshots: '@pkgr/core@0.3.6': {} - '@rolldown/binding-android-arm64@1.1.5': + '@rolldown/binding-android-arm64@1.2.3': optional: true - '@rolldown/binding-darwin-arm64@1.1.5': + '@rolldown/binding-darwin-arm64@1.2.3': optional: true - '@rolldown/binding-darwin-x64@1.1.5': + '@rolldown/binding-darwin-x64@1.2.3': optional: true - '@rolldown/binding-freebsd-x64@1.1.5': + '@rolldown/binding-freebsd-x64@1.2.3': optional: true - '@rolldown/binding-linux-arm-gnueabihf@1.1.5': + '@rolldown/binding-linux-arm-gnueabihf@1.2.3': optional: true - '@rolldown/binding-linux-arm64-gnu@1.1.5': + '@rolldown/binding-linux-arm64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-arm64-musl@1.1.5': + '@rolldown/binding-linux-arm64-musl@1.2.3': optional: true - '@rolldown/binding-linux-ppc64-gnu@1.1.5': + '@rolldown/binding-linux-ppc64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-s390x-gnu@1.1.5': + '@rolldown/binding-linux-s390x-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-gnu@1.1.5': + '@rolldown/binding-linux-x64-gnu@1.2.3': optional: true - '@rolldown/binding-linux-x64-musl@1.1.5': + '@rolldown/binding-linux-x64-musl@1.2.3': optional: true - '@rolldown/binding-openharmony-arm64@1.1.5': + '@rolldown/binding-openharmony-arm64@1.2.3': optional: true - '@rolldown/binding-wasm32-wasi@1.1.5': - dependencies: - '@emnapi/core': 1.11.1 - '@emnapi/runtime': 1.11.1 - '@napi-rs/wasm-runtime': 1.2.2(@emnapi/core@1.11.1)(@emnapi/runtime@1.11.1) - optional: true - - '@rolldown/binding-win32-arm64-msvc@1.1.5': + '@rolldown/binding-win32-arm64-msvc@1.2.3': optional: true - '@rolldown/binding-win32-x64-msvc@1.1.5': + '@rolldown/binding-win32-x64-msvc@1.2.3': optional: true '@rolldown/pluginutils@1.0.1': {} @@ -3728,15 +3674,15 @@ snapshots: picocolors: 1.1.1 redent: 3.0.0 - '@testing-library/react@16.3.2(@testing-library/dom@10.4.1)(@types/react-dom@19.2.3(@types/react@19.2.17))(@types/react@19.2.17)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': + '@testing-library/react@16.3.2(@testing-library/dom@10.4.1)(@types/react-dom@19.2.4(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)': dependencies: '@babel/runtime': 7.28.6 '@testing-library/dom': 10.4.1 react: 19.2.8 react-dom: 19.2.8(react@19.2.8) optionalDependencies: - '@types/react': 19.2.17 - '@types/react-dom': 19.2.3(@types/react@19.2.17) + '@types/react': 19.2.18 + '@types/react-dom': 19.2.4(@types/react@19.2.18) '@trivago/prettier-plugin-sort-imports@6.0.2(prettier@3.9.6)': dependencies: @@ -3752,11 +3698,6 @@ snapshots: transitivePeerDependencies: - supports-color - '@tybys/wasm-util@0.10.3': - dependencies: - tslib: 2.8.1 - optional: true - '@types/argparse@1.0.38': {} '@types/aria-query@5.0.4': {} @@ -3783,11 +3724,11 @@ snapshots: '@types/parse-json@4.0.2': {} - '@types/react-dom@19.2.3(@types/react@19.2.17)': + '@types/react-dom@19.2.4(@types/react@19.2.18)': dependencies: - '@types/react': 19.2.17 + '@types/react': 19.2.18 - '@types/react@19.2.17': + '@types/react@19.2.18': dependencies: csstype: 3.2.3 @@ -3888,11 +3829,11 @@ snapshots: '@typescript-eslint/types': 8.65.0 eslint-visitor-keys: 5.0.1 - '@vitejs/plugin-react-swc@4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3))': + '@vitejs/plugin-react-swc@4.3.3(@swc/helpers@0.5.23)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3))': dependencies: '@rolldown/pluginutils': 1.0.1 '@swc/core': 1.15.47(@swc/helpers@0.5.23) - vite: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + vite: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) transitivePeerDependencies: - '@swc/helpers' @@ -3911,7 +3852,7 @@ snapshots: std-env: 3.10.0 test-exclude: 7.0.2 tinyrainbow: 2.0.0 - vitest: 4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + vitest: 4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) transitivePeerDependencies: - supports-color @@ -3924,13 +3865,13 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.1 - '@vitest/mocker@4.1.10(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3))': + '@vitest/mocker@4.1.10(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3))': dependencies: '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + vite: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) '@vitest/pretty-format@4.1.10': dependencies: @@ -5266,7 +5207,7 @@ snapshots: globals@15.15.0: {} - globals@17.8.0: {} + globals@17.9.0: {} globalthis@1.0.4: dependencies: @@ -5675,7 +5616,7 @@ snapshots: muggle-string@0.4.1: {} - nanoid@3.3.16: {} + nanoid@3.3.17: {} natural-compare@1.4.0: {} @@ -5833,9 +5774,9 @@ snapshots: possible-typed-array-names@1.1.0: {} - postcss@8.5.25: + postcss@8.5.26: dependencies: - nanoid: 3.3.16 + nanoid: 3.3.17 picocolors: 1.1.1 source-map-js: 1.2.1 @@ -5943,26 +5884,25 @@ snapshots: path-parse: 1.0.7 supports-preserve-symlinks-flag: 1.0.0 - rolldown@1.1.5: + rolldown@1.2.3: dependencies: - '@oxc-project/types': 0.139.0 + '@oxc-project/types': 0.143.0 '@rolldown/pluginutils': 1.0.1 optionalDependencies: - '@rolldown/binding-android-arm64': 1.1.5 - '@rolldown/binding-darwin-arm64': 1.1.5 - '@rolldown/binding-darwin-x64': 1.1.5 - '@rolldown/binding-freebsd-x64': 1.1.5 - '@rolldown/binding-linux-arm-gnueabihf': 1.1.5 - '@rolldown/binding-linux-arm64-gnu': 1.1.5 - '@rolldown/binding-linux-arm64-musl': 1.1.5 - '@rolldown/binding-linux-ppc64-gnu': 1.1.5 - '@rolldown/binding-linux-s390x-gnu': 1.1.5 - '@rolldown/binding-linux-x64-gnu': 1.1.5 - '@rolldown/binding-linux-x64-musl': 1.1.5 - '@rolldown/binding-openharmony-arm64': 1.1.5 - '@rolldown/binding-wasm32-wasi': 1.1.5 - '@rolldown/binding-win32-arm64-msvc': 1.1.5 - '@rolldown/binding-win32-x64-msvc': 1.1.5 + '@rolldown/binding-android-arm64': 1.2.3 + '@rolldown/binding-darwin-arm64': 1.2.3 + '@rolldown/binding-darwin-x64': 1.2.3 + '@rolldown/binding-freebsd-x64': 1.2.3 + '@rolldown/binding-linux-arm-gnueabihf': 1.2.3 + '@rolldown/binding-linux-arm64-gnu': 1.2.3 + '@rolldown/binding-linux-arm64-musl': 1.2.3 + '@rolldown/binding-linux-ppc64-gnu': 1.2.3 + '@rolldown/binding-linux-s390x-gnu': 1.2.3 + '@rolldown/binding-linux-x64-gnu': 1.2.3 + '@rolldown/binding-linux-x64-musl': 1.2.3 + '@rolldown/binding-openharmony-arm64': 1.2.3 + '@rolldown/binding-win32-arm64-msvc': 1.2.3 + '@rolldown/binding-win32-x64-msvc': 1.2.3 rollup@4.59.0: dependencies: @@ -6327,11 +6267,11 @@ snapshots: spdx-correct: 3.2.0 spdx-expression-parse: 3.0.1 - vite-plugin-css-injected-by-js@4.0.1(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)): + vite-plugin-css-injected-by-js@4.0.1(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)): dependencies: - vite: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + vite: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) - vite-plugin-dts@4.5.4(@types/node@25.9.5)(rollup@4.59.0)(typescript@5.9.3)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)): + vite-plugin-dts@4.5.4(@types/node@25.9.5)(rollup@4.59.0)(typescript@5.9.3)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)): dependencies: '@microsoft/api-extractor': 7.57.2(@types/node@25.9.5) '@rollup/pluginutils': 5.3.0(rollup@4.59.0) @@ -6344,28 +6284,28 @@ snapshots: magic-string: 0.30.21 typescript: 5.9.3 optionalDependencies: - vite: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + vite: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) transitivePeerDependencies: - '@types/node' - rollup - supports-color - vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3): + vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3): dependencies: lightningcss: 1.33.0 picomatch: 4.0.5 - postcss: 8.5.25 - rolldown: 1.1.5 + postcss: 8.5.26 + rolldown: 1.2.3 tinyglobby: 0.2.17 optionalDependencies: '@types/node': 25.9.5 fsevents: 2.3.3 yaml: 2.8.3 - vitest@4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)): + vitest@4.1.10(@types/node@25.9.5)(@vitest/coverage-v8@3.2.7)(happy-dom@20.11.1)(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)): dependencies: '@vitest/expect': 4.1.10 - '@vitest/mocker': 4.1.10(vite@8.1.5(@types/node@25.9.5)(yaml@2.8.3)) + '@vitest/mocker': 4.1.10(vite@8.2.0(@types/node@25.9.5)(yaml@2.8.3)) '@vitest/pretty-format': 4.1.10 '@vitest/runner': 4.1.10 '@vitest/snapshot': 4.1.10 @@ -6382,7 +6322,7 @@ snapshots: tinyexec: 1.2.4 tinyglobby: 0.2.17 tinyrainbow: 3.1.1 - vite: 8.1.5(@types/node@25.9.5)(yaml@2.8.3) + vite: 8.2.0(@types/node@25.9.5)(yaml@2.8.3) why-is-node-running: 2.3.0 optionalDependencies: '@types/node': 25.9.5 diff --git a/dev/registry/extract_metadata.py b/dev/registry/extract_metadata.py index 1c557a3a8facb..77d34d8397d46 100644 --- a/dev/registry/extract_metadata.py +++ b/dev/registry/extract_metadata.py @@ -373,7 +373,9 @@ class Provider: } ) categories: list[dict] = field(default_factory=list) - connection_types: list[dict] = field(default_factory=list) # {conn_type, hook_class, docs_url} + connection_types: list[dict] = field( + default_factory=list + ) # {conn_type, hook_class, docs_url, external_services} requires_python: str = "" # e.g., ">=3.10" dependencies: list[str] = field(default_factory=list) # from pyproject.toml optional_extras: dict[str, list[str]] = field(default_factory=dict) # {extra_name: [deps]} @@ -782,12 +784,14 @@ def main(): for conn in provider_yaml.get("connection-types", []): conn_type = conn.get("connection-type", "") hook_class = conn.get("hook-class-name", "") + external_services = conn.get("external-services", []) if conn_type: connection_types.append( { "conn_type": conn_type, "hook_class": hook_class, "docs_url": resolve_connection_docs_url(conn_type, conn_url_map, base_docs_url), + "external_services": external_services, } ) diff --git a/dev/registry/extract_parameters.py b/dev/registry/extract_parameters.py index e32b06bae7ac7..7a44c4614ca6f 100644 --- a/dev/registry/extract_parameters.py +++ b/dev/registry/extract_parameters.py @@ -54,7 +54,13 @@ import yaml from extract_metadata import fetch_provider_inventory, read_inventory from registry_contract_models import validate_modules_catalog, validate_provider_parameters -from registry_tools.types import BASE_CLASS_IMPORTS, CLASS_LEVEL_SECTIONS, MODULE_LEVEL_SECTIONS +from registry_tools.types import ( + BASE_CLASS_IMPORTS, + CLASS_LEVEL_CATEGORY_OVERRIDES, + CLASS_LEVEL_SECTIONS, + DICT_SHAPED_CLASS_LEVEL_SECTIONS, + MODULE_LEVEL_SECTIONS, +) AIRFLOW_ROOT = Path(__file__).parent.parent.parent SCRIPT_DIR = Path(__file__).parent @@ -354,9 +360,9 @@ def _should_skip_class(name: str) -> bool: return False -def _get_first_docstring_line(cls: type) -> str | None: +def _get_first_docstring_line(obj: object) -> str | None: """Return the first non-empty line of a class docstring, or None.""" - doc = getattr(cls, "__doc__", None) + doc = getattr(obj, "__doc__", None) if not doc: return None for line in doc.strip().splitlines(): @@ -426,6 +432,27 @@ def is_durable_capable(cls: type, resumable_mixin: type | None) -> bool: return "execute_resumable" in source +def _resolve_dotted_path(class_path: str) -> tuple[str, str, object] | None: + """Split a dotted ``module.name`` path and import ``name`` from that module. + + Returns ``None`` (without logging) if ``class_path`` has no dot, or ``None`` + (after logging a warning) if the module import fails. Otherwise returns + ``(module_path, name, obj)``, where ``obj`` may be ``None`` if the module + has no such attribute — callers decide how to treat a missing attribute. + """ + parts = class_path.rsplit(".", 1) + if len(parts) != 2: + return None + module_path, name = parts + try: + mod = importlib.import_module(module_path) + obj = getattr(mod, name, None) + except Exception: + log.warning("Could not import %s", class_path) + return None + return module_path, name, obj + + def discover_classes_from_provider( provider_yaml_path: Path, base_classes: dict[str, type], @@ -583,29 +610,14 @@ def make_entry( for class_path in provider_yaml.get(section_name, []): if not class_path or not isinstance(class_path, str): continue - parts = class_path.rsplit(".", 1) - if len(parts) != 2: - continue - module_path, class_name = parts - try: - mod = importlib.import_module(module_path) - candidate = getattr(mod, class_name, None) - except Exception: - log.warning("Could not import %s", class_path) + if (resolved := _resolve_dotted_path(class_path)) is None: continue + module_path, class_name, candidate = resolved if candidate is None or not inspect.isclass(candidate): log.warning("%s is not a class", class_path) continue cls = candidate - # Use section name as category for class-level entries - category_map = { - "notifications": "notifications", - "secrets-backends": "secrets", - "logging": "logging", - "executors": "executors", - } - discovered.append( make_entry( cls, @@ -613,29 +625,52 @@ def make_entry( module_type, class_path, module_path, - category=category_map.get(section_name, section_name), + category=CLASS_LEVEL_CATEGORY_OVERRIDES.get(section_name, section_name), + ) + ) + + # --- Dict-shaped class-level sections (plugins, dialects; each entry is a + # dict carrying an integration-name field plus a class-path field, see types.py) --- + for section_name, ( + module_type, + class_field, + integration_field, + ) in DICT_SHAPED_CLASS_LEVEL_SECTIONS.items(): + for entry in provider_yaml.get(section_name, []): + if not isinstance(entry, dict): + continue + if not (class_path := entry.get(class_field, "")): + continue + if (resolved := _resolve_dotted_path(class_path)) is None: + continue + module_path, class_name, candidate = resolved + if candidate is None or not inspect.isclass(candidate): + log.warning("%s is not a class", class_path) + continue + + discovered.append( + make_entry( + candidate, + class_name, + module_type, + class_path, + module_path, + integration=entry.get(integration_field, ""), + category=CLASS_LEVEL_CATEGORY_OVERRIDES.get(section_name, section_name), ) ) # --- Task decorators (class-name key in each entry) --- for decorator in provider_yaml.get("task-decorators", []): - class_path = decorator.get("class-name", "") - decorator_name = decorator.get("name", "") - if not class_path: - continue - parts = class_path.rsplit(".", 1) - if len(parts) != 2: + if not (class_path := decorator.get("class-name", "")): continue - module_path, func_name = parts - try: - mod = importlib.import_module(module_path) - obj = getattr(mod, func_name, None) - except Exception: - log.warning("Could not import %s", class_path) + if (resolved := _resolve_dotted_path(class_path)) is None: continue + module_path, func_name, obj = resolved if obj is None: continue + decorator_name = decorator.get("name", "") display_name = f"@task.{decorator_name}" if decorator_name else func_name docstring = _get_first_docstring_line(obj) if hasattr(obj, "__doc__") else None short_desc = docstring or f"Task decorator for {decorator_name or func_name}" diff --git a/dev/registry/extract_versions.py b/dev/registry/extract_versions.py index cc5f094c79f05..064fd4a8f8628 100644 --- a/dev/registry/extract_versions.py +++ b/dev/registry/extract_versions.py @@ -59,7 +59,13 @@ sys.exit(1) from extract_metadata import fetch_provider_inventory, read_connection_urls, resolve_connection_docs_url -from registry_tools.types import MODULE_LEVEL_SECTIONS, TYPE_SUFFIXES +from registry_tools.types import ( + CLASS_LEVEL_CATEGORY_OVERRIDES, + CLASS_LEVEL_SECTIONS, + DICT_SHAPED_CLASS_LEVEL_SECTIONS, + MODULE_LEVEL_SECTIONS, + TYPE_SUFFIXES, +) SCRIPT_DIR = Path(__file__).parent AIRFLOW_ROOT = Path(__file__).parent.parent.parent @@ -76,6 +82,23 @@ REGISTRY_DIR / "src" / "_data" / "providers.json", ] +# Description suffix for class-level (FQCN) sections, keyed by module type id. +# Kept local to this file (unlike category, which is shared via +# CLASS_LEVEL_CATEGORY_OVERRIDES) because extract_parameters.py has a real +# docstring to use instead and doesn't need a description suffix at all. +FQCN_DESC_SUFFIXES: dict[str, str] = { + "notifier": "notifier", + "secret": "secrets backend", + "logging": "log handler", + "executor": "executor", + "extra_link": "extra link", + "queue": "queue", + "auth_manager": "auth manager", + "db_manager": "db manager", + "plugin": "plugin", + "dialect": "dialect", +} + def build_provider_id_to_path_map() -> dict[str, str]: """Scan providers/ for provider.yaml to build provider_id -> directory_path mapping.""" @@ -300,15 +323,13 @@ def process_module(module_path: str, module_type: str, integration: str, categor if mp: process_module(mp, "transfer", source, get_category(source)) - # Class-level sections (full class paths, no source file parsing needed) - FQCN_SECTIONS: dict[str, tuple[str, str, str]] = { - # yaml_key: (module_type, category, description_suffix) - "notifications": ("notifier", "notifications", "notifier"), - "secrets-backends": ("secret", "secrets", "secrets backend"), - "logging": ("logging", "logging", "log handler"), - "executors": ("executor", "executors", "executor"), - } - for yaml_key, (mod_type, category, desc_suffix) in FQCN_SECTIONS.items(): + # Class-level sections (full class paths, no source file parsing needed). + # Iterating CLASS_LEVEL_SECTIONS keeps this in sync with the single + # source of truth in types.py; a section added there without a matching + # FQCN_DESC_SUFFIXES entry raises KeyError instead of silently skipping. + for yaml_key, mod_type in CLASS_LEVEL_SECTIONS.items(): + category = CLASS_LEVEL_CATEGORY_OVERRIDES.get(yaml_key, yaml_key) + desc_suffix = FQCN_DESC_SUFFIXES[mod_type] for class_path in provider_yaml.get(yaml_key, []): if not class_path: continue @@ -329,6 +350,38 @@ def process_module(module_path: str, module_type: str, integration: str, categor } ) + # Dict-shaped class-level sections (each entry is a dict carrying the + # class path under a section-specific field name, see types.py). + for yaml_key, ( + mod_type, + class_path_field, + _integration_field, + ) in DICT_SHAPED_CLASS_LEVEL_SECTIONS.items(): + category = CLASS_LEVEL_CATEGORY_OVERRIDES.get(yaml_key, yaml_key) + desc_suffix = FQCN_DESC_SUFFIXES[mod_type] + for entry in provider_yaml.get(yaml_key, []): + if not isinstance(entry, dict): + continue + class_path = entry.get(class_path_field, "") + if not class_path: + continue + parts = class_path.rsplit(".", 1) + if len(parts) != 2: + continue + mod_path, class_name = parts + api_ref = mod_path.replace(".", "/") + modules.append( + { + "name": class_name, + "type": mod_type, + "import_path": class_path, + "short_description": f"{class_name} {desc_suffix}", + "docs_url": f"{base_docs_url}/_api/{api_ref}/index.html#{class_path}", + "source_url": f"{base_source_url}/{api_ref}.py", + "category": category, + } + ) + return modules @@ -396,6 +449,7 @@ def extract_version_data( "conn_type": conn_type, "hook_class": ct.get("hook-class-name", ""), "docs_url": resolve_connection_docs_url(conn_type, conn_url_map, base_docs_url), + "external_services": ct.get("external-services", []), } ) diff --git a/dev/registry/pyproject.toml b/dev/registry/pyproject.toml index 2ceef36c66beb..6f21421596b92 100644 --- a/dev/registry/pyproject.toml +++ b/dev/registry/pyproject.toml @@ -45,7 +45,7 @@ dependencies = [ ] [dependency-groups] -dev = ["pytest"] +dev = ["pytest>=9.1.1"] [tool.hatch.build.targets.wheel] packages = ["registry_tools"] diff --git a/dev/registry/registry_contract_models.py b/dev/registry/registry_contract_models.py index 7295cfb403614..475b14d5d0fcc 100644 --- a/dev/registry/registry_contract_models.py +++ b/dev/registry/registry_contract_models.py @@ -52,6 +52,7 @@ class ConnectionTypeContract(BaseModel): conn_type: str hook_class: str = "" docs_url: str | None = None + external_services: list[str] = Field(default_factory=list) class ProviderContract(BaseModel): diff --git a/dev/registry/registry_tools/types.py b/dev/registry/registry_tools/types.py index cd91b70ed6502..8f88ffe6e9764 100644 --- a/dev/registry/registry_tools/types.py +++ b/dev/registry/registry_tools/types.py @@ -116,6 +116,48 @@ "label": "Retry Policies", "icon": "R", }, + "extra_link": { + "yaml_key": "extra-links", + "level": "flat", + "suffixes": [], + "label": "Extra Links", + "icon": "I", + }, + "queue": { + "yaml_key": "queues", + "level": "flat", + "suffixes": [], + "label": "Message Queues", + "icon": "Q", + }, + "plugin": { + "yaml_key": "plugins", + "level": "flat", + "suffixes": [], + "label": "Plugins", + "icon": "P", + }, + "auth_manager": { + "yaml_key": "auth-managers", + "level": "flat", + "suffixes": [], + "label": "Auth Managers", + "icon": "A", + }, + "db_manager": { + "yaml_key": "db-managers", + "level": "flat", + "suffixes": [], + "label": "DB Managers", + "icon": "M", + }, + "dialect": { + "yaml_key": "dialects", + "level": "flat", + "suffixes": [], + "label": "Dialects", + "icon": "D", + }, } # Runtime base class imports for issubclass checks (extract_parameters.py). @@ -147,11 +189,38 @@ # Class-level sections used by extract_parameters.py (subset of flat that # list full class paths rather than simple entries). +# +# "plugins" and "dialects" are also flat/class-path sections, but each entry is a +# dict (plugin-class / dialect-class-name) rather than a bare string, so they are +# handled by the generic dict-shaped loop driven by DICT_SHAPED_CLASS_LEVEL_SECTIONS +# below instead of this table. CLASS_LEVEL_SECTIONS: dict[str, str] = { "notifications": "notifier", "secrets-backends": "secret", "logging": "logging", "executors": "executor", + "extra-links": "extra_link", + "queues": "queue", + "auth-managers": "auth_manager", + "db-managers": "db_manager", +} + +# Maps yaml section key -> (type id, class-path field name, integration-name +# field name) for dict-shaped class-level sections (each yaml entry is a dict, +# not a bare class-path string). Shared by extract_versions.py and +# extract_parameters.py so both walk these sections with a single generic loop +# instead of one per section, and so the field names are defined exactly once. +DICT_SHAPED_CLASS_LEVEL_SECTIONS: dict[str, tuple[str, str, str]] = { + "plugins": ("plugin", "plugin-class", "name"), + "dialects": ("dialect", "dialect-class-name", "dialect-type"), +} + +# Maps yaml section key -> category string for class-level (FQCN) sections. +# Only lists yaml keys whose category differs from the key itself. Callers +# fall back to the yaml key via .get(section_name, section_name) for every +# other key, including ones like "notifications" where key == category. +CLASS_LEVEL_CATEGORY_OVERRIDES: dict[str, str] = { + "secrets-backends": "secrets", } # All type ids, ordered consistently. diff --git a/dev/registry/tests/test_extract_metadata.py b/dev/registry/tests/test_extract_metadata.py index e4e98fcdc6985..faf17f6f9c8c4 100644 --- a/dev/registry/tests/test_extract_metadata.py +++ b/dev/registry/tests/test_extract_metadata.py @@ -20,6 +20,7 @@ import http.client import json +import sys import textwrap from pathlib import Path from unittest.mock import MagicMock, patch @@ -36,6 +37,7 @@ find_latest_released_version, find_related_providers, load_release_tags, + main, module_path_to_file_path, parse_pyproject_toml, preserve_nonzero_downloads, @@ -828,3 +830,57 @@ def test_filter_preserves_order(self): filtered = [v for v in raw_versions if f"providers-{provider_id}/{v}" in release_tags] # Order from raw_versions is preserved; only the phantom is dropped assert filtered == ["9.26.0", "9.25.0", "9.24.0"] + + +# --------------------------------------------------------------------------- +# main() -- connection-types `external-services` propagation +# --------------------------------------------------------------------------- +class TestMainConnectionTypesExternalServices: + """The connection-types extraction loop lives inline in main() rather than + a standalone function, so this drives main() end-to-end (with network and + filesystem dependencies mocked/redirected) to prove `external-services` + from provider.yaml reaches the written providers.json, and round-trips + through the `ConnectionTypeContract` (extra="forbid") validation main() + already runs -- catching a key-name drift between provider.yaml, this + script, and registry_contract_models.py. + """ + + @patch("extract_metadata.fetch_provider_inventory", autospec=True, return_value=None) + @patch("extract_metadata.fetch_pypi_data_parallel", autospec=True, return_value={}) + @patch("extract_metadata.load_release_tags", autospec=True, return_value=set()) + def test_external_services_propagates_to_providers_json( + self, _load_release_tags, _fetch_pypi_data_parallel, _fetch_provider_inventory, tmp_path + ): + providers_dir = tmp_path / "providers" + provider_dir = providers_dir / "testprov" + provider_dir.mkdir(parents=True) + (provider_dir / "provider.yaml").write_text( + textwrap.dedent("""\ + name: Test Provider + description: A test provider. + versions: + - 1.0.0 + connection-types: + - connection-type: testconn + hook-class-name: airflow.providers.test.hooks.TestHook + external-services: + - openai + - anthropic + """) + ) + output_dir = tmp_path / "output" + script_dir = tmp_path / "script" + output_dir.mkdir() + script_dir.mkdir() + + with ( + patch("extract_metadata.PROVIDERS_DIR", providers_dir), + patch("extract_metadata.OUTPUT_DIR", output_dir), + patch("extract_metadata.SCRIPT_DIR", script_dir), + patch.object(sys, "argv", ["extract_metadata.py"]), + ): + main() + + written = json.loads((output_dir / "providers.json").read_text()) + provider = next(p for p in written["providers"] if p["id"] == "testprov") + assert provider["connection_types"][0]["external_services"] == ["openai", "anthropic"] diff --git a/dev/registry/tests/test_extract_parameters.py b/dev/registry/tests/test_extract_parameters.py index 2cdb09532299f..3ebdf2004a72b 100644 --- a/dev/registry/tests/test_extract_parameters.py +++ b/dev/registry/tests/test_extract_parameters.py @@ -30,6 +30,7 @@ Module, _get_source_line, _parse_requested_providers, + _resolve_dotted_path, _should_skip_class, compare_with_ast, discover_classes_from_provider, @@ -370,6 +371,18 @@ class FakeSecretBackend: __module__ = "airflow.providers.amazon.aws.secrets.secrets_manager" +class FakePlugin: + """S3 plugin.""" + + __module__ = "airflow.providers.amazon.aws.plugins.s3" + + +class FakeDialect: + """Redshift dialect.""" + + __module__ = "airflow.providers.amazon.aws.dialects.redshift" + + def _make_module(name: str, members: dict) -> types.ModuleType: """Create a fake module with given members.""" mod = types.ModuleType(name) @@ -411,6 +424,20 @@ def _make_module(name: str, members: dict) -> types.ModuleType: "executors": [ "airflow.providers.amazon.aws.executors.ecs.FakeExecutor", ], + "plugins": [ + { + "name": "AmazonS3Plugin", + "plugin-class": "airflow.providers.amazon.aws.plugins.s3.FakePlugin", + }, + "not-a-dict-entry", + ], + "dialects": [ + { + "dialect-type": "redshift", + "dialect-class-name": "airflow.providers.amazon.aws.dialects.redshift.FakeDialect", + }, + "not-a-dict-entry", + ], "task-decorators": [], } @@ -473,6 +500,14 @@ def _mock_import(self, module_name): "airflow.providers.amazon.aws.executors.ecs", {"FakeExecutor": FakeExecutor}, ), + "airflow.providers.amazon.aws.plugins.s3": _make_module( + "airflow.providers.amazon.aws.plugins.s3", + {"FakePlugin": FakePlugin}, + ), + "airflow.providers.amazon.aws.dialects.redshift": _make_module( + "airflow.providers.amazon.aws.dialects.redshift", + {"FakeDialect": FakeDialect}, + ), } if module_name in modules: return modules[module_name] @@ -513,6 +548,42 @@ def test_discovers_hook(self, provider_yaml_path, base_classes): assert len(hooks) == 1 assert hooks[0]["name"] == "FakeHook" + def test_discovers_plugin(self, provider_yaml_path, base_classes): + with ( + patch("extract_parameters.PROVIDERS_DIR", provider_yaml_path.parent.parent), + patch("extract_parameters.importlib.import_module", side_effect=self._mock_import), + ): + result = discover_classes_from_provider(provider_yaml_path, base_classes) + + plugins = [r for r in result if r["type"] == "plugin"] + assert len(plugins) == 1 + assert plugins[0]["name"] == "FakePlugin" + assert plugins[0]["category"] == "plugins" + + def test_discovers_dialect(self, provider_yaml_path, base_classes): + with ( + patch("extract_parameters.PROVIDERS_DIR", provider_yaml_path.parent.parent), + patch("extract_parameters.importlib.import_module", side_effect=self._mock_import), + ): + result = discover_classes_from_provider(provider_yaml_path, base_classes) + + dialects = [r for r in result if r["type"] == "dialect"] + assert len(dialects) == 1 + assert dialects[0]["name"] == "FakeDialect" + assert dialects[0]["category"] == "dialects" + + def test_skips_non_dict_plugin_and_dialect_entries(self, provider_yaml_path, base_classes): + """FAKE_PROVIDER_YAML's plugins/dialects each carry a bare-string entry + alongside the valid dict entry; it must be skipped, not raise.""" + with ( + patch("extract_parameters.PROVIDERS_DIR", provider_yaml_path.parent.parent), + patch("extract_parameters.importlib.import_module", side_effect=self._mock_import), + ): + result = discover_classes_from_provider(provider_yaml_path, base_classes) + + assert len([r for r in result if r["type"] == "plugin"]) == 1 + assert len([r for r in result if r["type"] == "dialect"]) == 1 + def test_filters_reexported_classes(self, provider_yaml_path, base_classes): """Classes where cls.__module__ != the module being scanned should be excluded.""" with ( @@ -1086,3 +1157,31 @@ def test_extra_whitespace_is_tolerated(self): def test_duplicate_providers_collapsed(self): assert _parse_requested_providers("amazon amazon google") == {"amazon", "google"} + + +class TestResolveDottedPath: + @pytest.mark.parametrize( + "class_path", + ( + "no_dot_here", + "this.module.does.not.exist.at.all.Foo", + ), + ) + def test__resolve_dotted_path_returns_none(self, class_path: str): + assert _resolve_dotted_path(class_path) is None + + @pytest.mark.parametrize( + ("class_path", "expected"), + ( + ( + "extract_parameters.ThisAttrDoesNotExist", + ("extract_parameters", "ThisAttrDoesNotExist", None), + ), + ( + "extract_parameters.get_category", + ("extract_parameters", "get_category", get_category), + ), + ), + ) + def test__resolve_dotted_path(self, class_path: str, expected: tuple[str, str, object]): + assert _resolve_dotted_path(class_path) == expected diff --git a/dev/registry/tests/test_extract_versions.py b/dev/registry/tests/test_extract_versions.py index 45a2e26a2eafd..061424eb91806 100644 --- a/dev/registry/tests/test_extract_versions.py +++ b/dev/registry/tests/test_extract_versions.py @@ -18,11 +18,18 @@ from __future__ import annotations +import textwrap +from unittest.mock import patch + +import pytest from extract_versions import ( AIRFLOW_ROOT, PROVIDERS_JSON_CANDIDATES, SCRIPT_DIR, + extract_modules_from_yaml, + extract_version_data, ) +from registry_tools.types import CLASS_LEVEL_SECTIONS, DICT_SHAPED_CLASS_LEVEL_SECTIONS class TestProvidersJsonCandidates: @@ -49,3 +56,157 @@ def test_no_other_candidates(self): # should be caught and fixed at the source. Match siblings (extract_ # parameters.py, extract_metadata.py) which use exactly these two. assert len(PROVIDERS_JSON_CANDIDATES) == 2 + + +# Expected (category, description_suffix) per class-level section, entered by +# hand from the PR #70190 spec rather than derived from production code, so a +# regression in extract_versions.py's mapping logic is actually caught. +EXPECTED_CLASS_LEVEL_CATEGORIES = { + "notifications": "notifications", + "secrets-backends": "secrets", + "logging": "logging", + "executors": "executors", + "extra-links": "extra-links", + "queues": "queues", + "auth-managers": "auth-managers", + "db-managers": "db-managers", +} + +EXPECTED_CLASS_LEVEL_DESC_SUFFIXES = { + "notifier": "notifier", + "secret": "secrets backend", + "logging": "log handler", + "executor": "executor", + "extra_link": "extra link", + "queue": "queue", + "auth_manager": "auth manager", + "db_manager": "db manager", +} + + +def _extract_class_level_modules(provider_yaml: dict) -> list[dict]: + return extract_modules_from_yaml( + provider_yaml, + tag="providers-test/1.0.0", + layout="new", + dir_path="test", + provider_id="test", + version="1.0.0", + ) + + +class TestExtractModulesFromYamlClassLevelSections: + @pytest.mark.parametrize(("yaml_key", "mod_type"), list(CLASS_LEVEL_SECTIONS.items())) + def test_class_level_section_produces_module(self, yaml_key, mod_type): + class_path = f"airflow.providers.test.{yaml_key.replace('-', '_')}.example.ExampleClass" + provider_yaml = {yaml_key: [class_path]} + + modules = _extract_class_level_modules(provider_yaml) + + assert len(modules) == 1 + module = modules[0] + assert module["type"] == mod_type + assert module["category"] == EXPECTED_CLASS_LEVEL_CATEGORIES[yaml_key] + expected_desc_suffix = EXPECTED_CLASS_LEVEL_DESC_SUFFIXES[mod_type] + assert module["short_description"] == f"ExampleClass {expected_desc_suffix}" + + def test_all_class_level_sections_produce_covered_types(self): + """Type coverage must track CLASS_LEVEL_SECTIONS; a key added there + without matching handling in extract_versions.py fails here.""" + provider_yaml = { + yaml_key: [f"airflow.providers.test.{yaml_key.replace('-', '_')}.example.ExampleClass"] + for yaml_key in CLASS_LEVEL_SECTIONS + } + + modules = _extract_class_level_modules(provider_yaml) + + assert {m["type"] for m in modules} == set(CLASS_LEVEL_SECTIONS.values()) + + +# plugins/dialects aren't in CLASS_LEVEL_CATEGORY_OVERRIDES, so category == +# yaml_key itself (same fallback as EXPECTED_CLASS_LEVEL_CATEGORIES above). +EXPECTED_DICT_SHAPED_CATEGORIES = { + "plugins": "plugins", + "dialects": "dialects", +} + +EXPECTED_DICT_SHAPED_DESC_SUFFIXES = { + "plugin": "plugin", + "dialect": "dialect", +} + + +class TestExtractModulesFromYamlDictShapedSections: + @pytest.mark.parametrize( + ("yaml_key", "type_and_field"), + list(DICT_SHAPED_CLASS_LEVEL_SECTIONS.items()), + ) + def test_dict_shaped_section_produces_module(self, yaml_key, type_and_field): + mod_type, class_path_field, _integration_field = type_and_field + class_path = f"airflow.providers.test.{yaml_key.replace('-', '_')}.example.ExampleClass" + provider_yaml = {yaml_key: [{class_path_field: class_path}]} + + modules = _extract_class_level_modules(provider_yaml) + + assert len(modules) == 1 + module = modules[0] + assert module["type"] == mod_type + assert module["category"] == EXPECTED_DICT_SHAPED_CATEGORIES[yaml_key] + expected_desc_suffix = EXPECTED_DICT_SHAPED_DESC_SUFFIXES[mod_type] + assert module["short_description"] == f"ExampleClass {expected_desc_suffix}" + + def test_all_dict_shaped_sections_produce_covered_types(self): + """Type coverage must track DICT_SHAPED_CLASS_LEVEL_SECTIONS; a key + added there without matching handling in extract_versions.py fails here.""" + provider_yaml = { + yaml_key: [ + { + class_path_field: f"airflow.providers.test.{yaml_key.replace('-', '_')}.example.ExampleClass" + } + ] + for yaml_key, ( + _, + class_path_field, + _integration_field, + ) in DICT_SHAPED_CLASS_LEVEL_SECTIONS.items() + } + + modules = _extract_class_level_modules(provider_yaml) + + assert {m["type"] for m in modules} == {t for t, _, _ in DICT_SHAPED_CLASS_LEVEL_SECTIONS.values()} + + +class TestExtractVersionDataConnectionTypes: + """`external-services` on a connection-types entry must survive into the + per-version metadata.json (extract_versions.py:399) the same way it does + for the latest release in providers.json (extract_metadata.py). A + superseded release only has this file as its data source, so a dropped + or mis-keyed field here silently vanishes from just that version's page. + """ + + PROVIDER_YAML = textwrap.dedent("""\ + name: Test Provider + connection-types: + - connection-type: testconn + hook-class-name: airflow.providers.test.hooks.TestHook + external-services: + - openai + - anthropic + """) + + @patch("extract_versions.extract_modules_from_yaml", autospec=True, return_value=[]) + @patch("extract_versions.fetch_provider_inventory", autospec=True, return_value=None) + @patch("extract_versions.git_show", autospec=True) + @patch("extract_versions.detect_layout", autospec=True, return_value="new") + @patch("extract_versions.git_tag_exists", autospec=True, return_value=True) + def test_external_services_propagates_to_version_metadata( + self, _tag_exists, _layout, mock_git_show, _inventory, _modules + ): + mock_git_show.side_effect = lambda tag, path: ( + self.PROVIDER_YAML if path.endswith("provider.yaml") else None + ) + + result = extract_version_data("test", "1.0.0", "test") + + assert result is not None + assert result["connection_types"][0]["external_services"] == ["openai", "anthropic"] diff --git a/dev/registry/tests/test_registry_contract_models.py b/dev/registry/tests/test_registry_contract_models.py index 120c682da78d7..2a98d84826123 100644 --- a/dev/registry/tests/test_registry_contract_models.py +++ b/dev/registry/tests/test_registry_contract_models.py @@ -21,6 +21,7 @@ import pytest from pydantic import ValidationError from registry_contract_models import ( + ConnectionTypeContract, build_openapi_document, validate_modules_catalog, validate_provider_parameters, @@ -100,6 +101,24 @@ def test_module_contract_preserves_supports_durable_execution_true(): assert validated["modules"][0]["supports_durable_execution"] is True +def test_connection_type_contract_defaults_external_services_to_empty_list(): + """Legacy connection-types entries (provider.yaml without `external-services`) + must still validate, with the field defaulting to an empty list.""" + validated = ConnectionTypeContract.model_validate({"conn_type": "test", "hook_class": "x.y.Hook"}) + assert validated.external_services == [] + + +def test_connection_type_contract_round_trips_external_services(): + validated = ConnectionTypeContract.model_validate( + { + "conn_type": "test", + "hook_class": "x.y.Hook", + "external_services": ["openai", "anthropic"], + } + ) + assert validated.external_services == ["openai", "anthropic"] + + def test_validate_version_metadata_accepts_legacy_version_modules_without_ids(): payload = { "provider_id": "test", diff --git a/dev/registry/tests/test_types.py b/dev/registry/tests/test_types.py index d5d2dfbf21870..85bb578745d0a 100644 --- a/dev/registry/tests/test_types.py +++ b/dev/registry/tests/test_types.py @@ -22,7 +22,9 @@ from registry_tools.types import ( ALL_TYPE_IDS, BASE_CLASS_IMPORTS, + CLASS_LEVEL_CATEGORY_OVERRIDES, CLASS_LEVEL_SECTIONS, + DICT_SHAPED_CLASS_LEVEL_SECTIONS, FLAT_LEVEL_SECTIONS, MODULE_LEVEL_SECTIONS, MODULE_TYPES, @@ -93,6 +95,30 @@ def test_class_level_sections_are_subset_of_flat(self): assert yaml_key in FLAT_LEVEL_SECTIONS assert FLAT_LEVEL_SECTIONS[yaml_key] == type_id + def test_class_level_category_overrides_are_subset_of_class_level_sections(self): + assert set(CLASS_LEVEL_CATEGORY_OVERRIDES.keys()) <= set(CLASS_LEVEL_SECTIONS.keys()) + + def test_dict_shaped_class_level_sections_are_subset_of_flat(self): + for yaml_key, (type_id, _field_name, _integration_field) in DICT_SHAPED_CLASS_LEVEL_SECTIONS.items(): + assert yaml_key in FLAT_LEVEL_SECTIONS + assert FLAT_LEVEL_SECTIONS[yaml_key] == type_id + + def test_every_flat_section_is_consumed_by_a_class_level_table(self): + # "transfers" and "task-decorators" are deliberately exempt: they are + # consumed by their own dedicated extraction paths rather than by + # CLASS_LEVEL_SECTIONS or DICT_SHAPED_CLASS_LEVEL_SECTIONS. + dedicated_path_exemptions = {"transfers", "task-decorators"} + consumed = ( + set(CLASS_LEVEL_SECTIONS) | set(DICT_SHAPED_CLASS_LEVEL_SECTIONS) | dedicated_path_exemptions + ) + assert set(FLAT_LEVEL_SECTIONS) == consumed, ( + f"set(FLAT_LEVEL_SECTIONS) {set(FLAT_LEVEL_SECTIONS)} != consumed {consumed}. " + "Every flat-level section must be consumed by CLASS_LEVEL_SECTIONS or " + "DICT_SHAPED_CLASS_LEVEL_SECTIONS, otherwise both extractors silently skip it. " + "Add the new section to whichever table matches its yaml shape, or, if it is " + "handled by a dedicated path, add it to dedicated_path_exemptions above." + ) + class TestBaseClassImports: def test_all_entries_are_tuples(self): diff --git a/dev/skill-evals/last-eval-hash.txt b/dev/skill-evals/last-eval-hash.txt index 8d29ed7d85ade..23452e4e6f3a0 100644 --- a/dev/skill-evals/last-eval-hash.txt +++ b/dev/skill-evals/last-eval-hash.txt @@ -1,3 +1,3 @@ # Generated by dev/skill-evals/eval.py — do not edit or resolve conflicts by hand. # Run `prek run run-skill-eval --hook-stage manual --all-files` to regenerate. -cdf346752bd43c958355d98e63453bc4a00dd82727dde4243b1c01a2e2afb5fe +725f2a0640c5b6fec7f6666484a0577dde5c87f19206b1d0bdf191a7094cd98d diff --git a/devel-common/pyproject.toml b/devel-common/pyproject.toml index f285c3744a7c3..a4ece147a0b57 100644 --- a/devel-common/pyproject.toml +++ b/devel-common/pyproject.toml @@ -34,7 +34,7 @@ dependencies = [ "kgb>=7.2.0", "requests_mock>=1.11.0", "rich>=13.6.0", - "ruff==0.15.22", + "ruff==0.16.0", "semver>=3.0.2", "typer>=0.22.0", "time-machine[dateutil]>=3.0.0", @@ -118,7 +118,7 @@ dependencies = [ "types-certifi>=2021.10.8.3", "types-croniter>=2.0.0.20240423", "types-docutils>=0.21.0.20240704", - "types-paramiko>=4.0.0.20260402,<5.0.0", + "types-paramiko>=4.0.0.20260402", "types-protobuf>=5.26.0.20240422", "types-python-dateutil>=2.9.0.20240316", "types-python-slugify>=8.0.2.20240310", diff --git a/generated/known_airflow_exceptions.txt b/generated/known_airflow_exceptions.txt index e170208c3678d..b9615408a9942 100644 --- a/generated/known_airflow_exceptions.txt +++ b/generated/known_airflow_exceptions.txt @@ -199,8 +199,7 @@ providers/fab/src/airflow/providers/fab/auth_manager/fab_auth_manager.py::3 providers/fab/src/airflow/providers/fab/auth_manager/models/db.py::1 providers/fab/src/airflow/providers/fab/www/extensions/init_security.py::1 providers/facebook/src/airflow/providers/facebook/ads/hooks/ads.py::2 -providers/git/src/airflow/providers/git/bundles/git.py::5 -providers/git/src/airflow/providers/git/hooks/git.py::1 +providers/git/src/airflow/providers/git/bundles/git.py::4 providers/git/tests/unit/git/bundles/test_git.py::1 providers/github/src/airflow/providers/github/operators/github.py::2 providers/github/src/airflow/providers/github/sensors/github.py::3 diff --git a/generated/provider_metadata.json b/generated/provider_metadata.json index c1ce7da75000e..c152712e4828d 100644 --- a/generated/provider_metadata.json +++ b/generated/provider_metadata.json @@ -175,6 +175,10 @@ "6.0.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.0.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:12Z" } }, "akeyless": { @@ -371,6 +375,10 @@ "3.3.9": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-22T21:14:32Z" + }, + "3.4.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:12Z" } }, "amazon": { @@ -769,6 +777,10 @@ "9.33.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "9.34.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:13Z" } }, "anthropic": { @@ -779,6 +791,10 @@ "0.2.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "0.2.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:14Z" } }, "apache.beam": { @@ -1771,6 +1787,10 @@ "4.12.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-22T21:14:32Z" + }, + "4.12.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:15Z" } }, "apache.hive": { @@ -2049,6 +2069,10 @@ "9.6.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "9.6.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:16Z" } }, "apache.iceberg": { @@ -2359,6 +2383,10 @@ "1.15.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "1.15.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:17Z" } }, "apache.kylin": { @@ -2687,6 +2715,10 @@ "4.5.7": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-22T21:14:32Z" + }, + "4.6.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:18Z" } }, "apache.pig": { @@ -3213,6 +3245,10 @@ "6.3.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.3.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:19Z" } }, "apache.tinkerpop": { @@ -3475,6 +3511,10 @@ "2.9.5": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-07T10:04:12Z" + }, + "2.9.6": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:20Z" } }, "asana": { @@ -3977,6 +4017,10 @@ "3.23.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "3.23.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:21Z" } }, "clickhousedb": { @@ -4625,6 +4669,10 @@ "1.6.6": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-07T10:04:16Z" + }, + "1.6.7": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:22Z" } }, "common.ai": { @@ -4655,6 +4703,10 @@ "0.6.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:24Z" + }, + "0.7.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:23Z" } }, "common.compat": { @@ -4777,6 +4829,10 @@ "1.17.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "1.18.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:24Z" } }, "common.io": { @@ -5203,6 +5259,10 @@ "2.0.3": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "2.1.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:25Z" } }, "databricks": { @@ -5517,6 +5577,10 @@ "7.18.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "7.18.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:26Z" } }, "datadog": { @@ -5881,6 +5945,10 @@ "4.9.2": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-22T21:14:32Z" + }, + "4.9.3": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:27Z" } }, "dingding": { @@ -6411,6 +6479,10 @@ "4.5.8": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "4.5.9": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:28Z" } }, "edge3": { @@ -6525,6 +6597,10 @@ "4.2.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "4.3.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:29Z" } }, "elasticsearch": { @@ -6795,6 +6871,10 @@ "6.8.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.9.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:30Z" } }, "exasol": { @@ -7009,6 +7089,10 @@ "4.10.4": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:25Z" + }, + "4.10.5": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:31Z" } }, "fab": { @@ -7211,6 +7295,10 @@ "3.7.3": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "3.8.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:32Z" } }, "facebook": { @@ -7613,6 +7701,10 @@ "0.4.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:25Z" + }, + "0.4.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:33Z" } }, "github": { @@ -8457,6 +8549,10 @@ "4.7.2": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "4.8.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:34Z" } }, "http": { @@ -8853,6 +8949,10 @@ "3.12.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:25Z" + }, + "3.12.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:35Z" } }, "influxdb": { @@ -9817,6 +9917,10 @@ "14.0.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "14.1.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:36Z" } }, "microsoft.mssql": { @@ -10897,6 +11001,10 @@ "3.12.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:24Z" + }, + "3.12.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:37Z" } }, "odbc": { @@ -11173,6 +11281,10 @@ "1.8.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "1.8.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:38Z" } }, "openfaas": { @@ -11517,6 +11629,10 @@ "2.19.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:24Z" + }, + "2.20.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:39Z" } }, "opensearch": { @@ -11647,6 +11763,10 @@ "1.11.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "1.12.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:40Z" } }, "opsgenie": { @@ -12335,6 +12455,10 @@ "3.13.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-07T10:04:52Z" + }, + "3.13.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:41Z" } }, "pgvector": { @@ -12769,6 +12893,10 @@ "7.0.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "7.0.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:42Z" } }, "presto": { @@ -12983,6 +13111,10 @@ "5.12.0": { "associated_airflow_version": "3.2.2", "date_released": "2026-05-23T12:35:14Z" + }, + "5.12.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:43Z" } }, "qdrant": { @@ -13409,6 +13541,10 @@ "5.14.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-07T10:04:57Z" + }, + "5.14.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:44Z" } }, "samba": { @@ -14065,6 +14201,10 @@ "6.0.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.0.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:45Z" } }, "singularity": { @@ -14579,6 +14719,10 @@ "3.0.2": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:25Z" + }, + "3.0.3": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:46Z" } }, "snowflake": { @@ -14913,6 +15057,10 @@ "6.15.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.16.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:47Z" } }, "sqlite": { @@ -15317,6 +15465,10 @@ "6.0.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "6.0.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:48Z" } }, "standard": { @@ -15463,6 +15615,10 @@ "1.16.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-10T16:30:24Z" + }, + "1.17.0": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:49Z" } }, "tableau": { @@ -15787,6 +15943,10 @@ "4.9.5": { "associated_airflow_version": "3.3.0", "date_released": "2026-06-07T10:05:08Z" + }, + "4.9.6": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:50Z" } }, "teradata": { @@ -15905,6 +16065,10 @@ "3.6.1": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "3.6.2": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:51Z" } }, "trino": { @@ -16135,6 +16299,10 @@ "6.6.0": { "associated_airflow_version": "3.2.2", "date_released": "2026-05-23T12:35:14Z" + }, + "6.6.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:52Z" } }, "vertica": { @@ -16429,6 +16597,10 @@ "3.4.0": { "associated_airflow_version": "3.3.0", "date_released": "2026-07-28T06:43:08Z" + }, + "3.4.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:53Z" } }, "yandex": { @@ -16591,6 +16763,10 @@ "4.5.0": { "associated_airflow_version": "3.2.2", "date_released": "2026-05-11T15:47:09Z" + }, + "4.5.1": { + "associated_airflow_version": "3.3.0", + "date_released": "2026-08-08T06:51:54Z" } }, "ydb": { diff --git a/providers-summary-docs/howto/create-custom-providers.rst b/providers-summary-docs/howto/create-custom-providers.rst index 68d2da6acdd05..ed6b00b9a663e 100644 --- a/providers-summary-docs/howto/create-custom-providers.rst +++ b/providers-summary-docs/howto/create-custom-providers.rst @@ -114,7 +114,7 @@ Exposing customized functionality to the Airflow's core: * ``sensors`` - this field should contain the list of all the sensor class names that the provider provides. See :doc:`apache-airflow:core-concepts/sensors` for description of the sensors. -* ``task-decorators`` - this field should contain the list of dictionaries of name/path where the decorators +* ``task-decorators`` - this field should contain the list of dictionaries of name/class-name where the decorators are available. See :doc:`apache-airflow:howto/create-custom-decorator` for description of how to add custom decorators. diff --git a/providers/.last_release_date.txt b/providers/.last_release_date.txt index 9c6e373bfef6c..56be2e8dc64c2 100644 --- a/providers/.last_release_date.txt +++ b/providers/.last_release_date.txt @@ -1 +1 @@ -2026-08-01 +2026-08-08 diff --git a/providers/akeyless/docs/index.rst b/providers/akeyless/docs/index.rst index ec6bbd5f176c3..8e18b2cefc92a 100644 --- a/providers/akeyless/docs/index.rst +++ b/providers/akeyless/docs/index.rst @@ -117,11 +117,11 @@ Install them when installing from PyPI. For example: pip install apache-airflow-providers-akeyless[cloud_id] -============ ===================== +============ ============================ Extra Dependencies -============ ===================== -``cloud_id`` ``akeyless_cloud_id`` -============ ===================== +============ ============================ +``cloud_id`` ``akeyless-cloud-id>=0.3.0`` +============ ============================ Downloading official packages ----------------------------- diff --git a/providers/akeyless/pyproject.toml b/providers/akeyless/pyproject.toml index f9e70fdf01a02..b4b79d90f2829 100644 --- a/providers/akeyless/pyproject.toml +++ b/providers/akeyless/pyproject.toml @@ -68,7 +68,7 @@ dependencies = [ # Any change in the dependencies is preserved when the file is regenerated [project.optional-dependencies] "cloud_id" = [ - "akeyless_cloud_id", + "akeyless-cloud-id>=0.3.0", ] [dependency-groups] diff --git a/providers/amazon/README.rst b/providers/amazon/README.rst index 51d9b03452f44..4e5804faa02c2 100644 --- a/providers/amazon/README.rst +++ b/providers/amazon/README.rst @@ -105,9 +105,9 @@ Dependent package Optional dependencies ---------------------- -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ Extra Dependencies -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ ``aiobotocore`` ``aiobotocore>=3.0.0`` ``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.2.0`` ``s3fs`` ``s3fs>=2023.10.0`` @@ -120,14 +120,14 @@ Extra Dependencies ``imap`` ``apache-airflow-providers-imap`` ``microsoft.azure`` ``apache-airflow-providers-microsoft-azure`` ``mongo`` ``apache-airflow-providers-mongo`` -``pandas`` ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``openlineage`` ``apache-airflow-providers-openlineage>=2.3.0`` ``salesforce`` ``apache-airflow-providers-salesforce`` ``ssh`` ``apache-airflow-providers-ssh`` ``standard`` ``apache-airflow-providers-standard`` ``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/amazon/docs/changelog.rst b/providers/amazon/docs/changelog.rst index b8c0691e90ca7..a63a7bdd07807 100644 --- a/providers/amazon/docs/changelog.rst +++ b/providers/amazon/docs/changelog.rst @@ -52,11 +52,11 @@ Bug Fixes * ``Strip GlueDataQualityOperator ruleset after rendering (#70331)`` * ``Move template-field validation/transformation out of init for Amazon Bedrock operators (#70306)`` * ``Move template-field validation out of init for Amazon DataSync operator (#70324)`` +* ``Only refuse team scoped like secret ids when multi_team is on (#71078)`` Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Keep S3DeleteObjectsOperator validation in __init__, narrow to ValueError (#70359)`` * ``Rename StepFunctionStartExecutionOperator's input field to match its constructor argument (#70544)`` * ``Fix mypy arg-type error in Redshift Data get_primary_keys (#70677)`` @@ -77,6 +77,10 @@ Doc-only * ``Make Redshift system-test clusters non-public in sql_to_s3/s3_to_sql (#70705)`` * ``Wait for vector index to stabilize in 'example_bedrock_retrieve_and_generate' (#70649)`` * ``Fix flaky Glue OpenLineage test that calls the real AWS endpoint (#70500)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` + * ``Add support for pandas 3 based xcoms in airflow (#71103)`` + * ``Fix some docs and test gaps following up multi team secret refusal (#71106)`` 9.33.0 ...... @@ -1324,7 +1328,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 9.3.0 ..... diff --git a/providers/amazon/docs/index.rst b/providers/amazon/docs/index.rst index 73635c6f4cebd..7dc10574bd41c 100644 --- a/providers/amazon/docs/index.rst +++ b/providers/amazon/docs/index.rst @@ -171,9 +171,9 @@ Install them when installing from PyPI. For example: pip install apache-airflow-providers-amazon[aiobotocore] -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ Extra Dependencies -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ ``aiobotocore`` ``aiobotocore>=3.0.0`` ``cncf.kubernetes`` ``apache-airflow-providers-cncf-kubernetes>=7.2.0`` ``s3fs`` ``s3fs>=2023.10.0`` @@ -186,14 +186,14 @@ Extra Dependencies ``imap`` ``apache-airflow-providers-imap`` ``microsoft.azure`` ``apache-airflow-providers-microsoft-azure`` ``mongo`` ``apache-airflow-providers-mongo`` -``pandas`` ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``openlineage`` ``apache-airflow-providers-openlineage>=2.3.0`` ``salesforce`` ``apache-airflow-providers-salesforce`` ``ssh`` ``apache-airflow-providers-ssh`` ``standard`` ``apache-airflow-providers-standard`` ``common.messaging`` ``apache-airflow-providers-common-messaging>=2.0.0`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` -==================== ===================================================================================================================================================================== +==================== ============================================================================================================================================================ Downloading official packages ----------------------------- diff --git a/providers/amazon/pyproject.toml b/providers/amazon/pyproject.toml index a5d46203f5779..25d1d6320bc4c 100644 --- a/providers/amazon/pyproject.toml +++ b/providers/amazon/pyproject.toml @@ -130,18 +130,9 @@ dependencies = [ "apache-airflow-providers-mongo" ] "pandas" = [ - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] "openlineage" = [ "apache-airflow-providers-openlineage>=2.3.0" diff --git a/providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py b/providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py index 415fc94591081..bf17d613238be 100644 --- a/providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py +++ b/providers/amazon/src/airflow/providers/amazon/aws/secrets/secrets_manager.py @@ -25,6 +25,7 @@ from typing import Any from airflow.providers.amazon.aws.utils import trim_none_values +from airflow.providers.common.compat.sdk import conf from airflow.secrets import BaseSecretsBackend from airflow.utils.log.logging_mixin import LoggingMixin @@ -335,7 +336,12 @@ def _names_a_team_namespace(secret_id: str) -> bool: the caller's own team builds looks equivalent and is not -- a caller in team ``a`` would match ``a--b``'s namespace on the prefix and read its secrets. Only the caller's own namespace is ever constructed, never parsed. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped secret can exist to collide with. """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False return TEAM_SEP in secret_id def _log_refusal(self, kind: str, secret_id: str) -> None: diff --git a/providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py b/providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py index 5cae97670b7b0..8563d5ce9f695 100644 --- a/providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py +++ b/providers/amazon/src/airflow/providers/amazon/aws/secrets/systems_manager.py @@ -23,6 +23,7 @@ from functools import cached_property from airflow.providers.amazon.aws.utils import trim_none_values +from airflow.providers.common.compat.sdk import conf from airflow.secrets import BaseSecretsBackend from airflow.utils.log.logging_mixin import LoggingMixin @@ -214,7 +215,12 @@ def _names_a_team_namespace(secret_id: str) -> bool: the caller's own team builds looks equivalent and is not -- a caller in team ``a`` would match ``a--b``'s namespace on the prefix and read its secrets. Only the caller's own namespace is ever constructed, never parsed. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped secret can exist to collide with. """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False return TEAM_SEP in secret_id def _log_refusal(self, kind: str, secret_id: str) -> None: diff --git a/providers/amazon/src/airflow/providers/amazon/aws/sensors/bedrock.py b/providers/amazon/src/airflow/providers/amazon/aws/sensors/bedrock.py index 7ccac1f87eb93..5d00918ec46fe 100644 --- a/providers/amazon/src/airflow/providers/amazon/aws/sensors/bedrock.py +++ b/providers/amazon/src/airflow/providers/amazon/aws/sensors/bedrock.py @@ -79,14 +79,27 @@ def __init__( def poke(self, context: Context, **kwargs) -> bool: state = self.get_state() if state in self.FAILURE_STATES: - raise AirflowException(self.FAILURE_MESSAGE) + raise AirflowException(f"{self.FAILURE_MESSAGE}{self._failure_reason_suffix()}") return state not in self.INTERMEDIATE_STATES + def _failure_reason_suffix(self) -> str: + """Assemble failure reasons if any are provided.""" + return f" Failure reasons: {reason}" if (reason := self.get_failure_reason()) else "" + @abc.abstractmethod def get_state(self) -> str: """Implement in subclasses.""" + def get_failure_reason(self) -> str: + """ + Return the service-reported reason(s) for a failed state, or an empty string if none. + + Override in subclasses whose describe API exposes failure detail. Return the reasons + only; phrasing and separators are handled by the caller. + """ + return "" + class BedrockCustomizeModelCompletedSensor(BedrockBaseSensor[BedrockHook]): """ @@ -348,12 +361,20 @@ def __init__( self.data_source_id = data_source_id self.ingestion_job_id = ingestion_job_id - def get_state(self) -> str: + def _get_ingestion_job(self) -> dict[str, Any]: return self.hook.conn.get_ingestion_job( knowledgeBaseId=self.knowledge_base_id, ingestionJobId=self.ingestion_job_id, dataSourceId=self.data_source_id, - )["ingestionJob"]["status"] + )["ingestionJob"] + + def get_state(self) -> str: + return self._get_ingestion_job()["status"] + + def get_failure_reason(self) -> str: + if reasons := self._get_ingestion_job().get("failureReasons"): + return "; ".join(reasons) + return "" def execute(self, context: Context) -> Any: if self.deferrable: diff --git a/providers/amazon/tests/system/amazon/aws/example_emr_eks.py b/providers/amazon/tests/system/amazon/aws/example_emr_eks.py index 56af4e1f931fa..ef2423acc822b 100644 --- a/providers/amazon/tests/system/amazon/aws/example_emr_eks.py +++ b/providers/amazon/tests/system/amazon/aws/example_emr_eks.py @@ -16,14 +16,12 @@ # under the License. from __future__ import annotations -import json import logging +import re import subprocess -import time from datetime import datetime, timedelta import boto3 -from tenacity import retry, retry_if_exception_type, stop_after_delay, wait_exponential from airflow.providers.amazon.aws.hooks.eks import ClusterStates, NodegroupStates from airflow.providers.amazon.aws.operators.eks import EksCreateClusterOperator, EksDeleteClusterOperator @@ -71,6 +69,9 @@ .build() ) +# `emr-containers create-role-associations` / `delete-role-associations` were added in this release. +MIN_AWS_CLI_VERSION = (2, 24, 0) + S3_FILE_NAME = "pi.py" S3_FILE_CONTENT = """ k = 1 @@ -111,8 +112,7 @@ def run_eksctl_commands(cluster_name, ns): file = "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" commands = f""" curl --silent --location "{file}" | tar xz -C /tmp && - /tmp/eksctl create iamidentitymapping --cluster {cluster_name} --namespace {ns} --service-name "emr-containers" && - /tmp/eksctl utils associate-iam-oidc-provider --cluster {cluster_name} --approve + /tmp/eksctl create iamidentitymapping --cluster {cluster_name} --namespace {ns} --service-name "emr-containers" """ build = subprocess.Popen( @@ -127,43 +127,75 @@ def run_eksctl_commands(cluster_name, ns): raise RuntimeError(err) -@task(trigger_rule=TriggerRule.ALL_DONE) -def delete_iam_oidc_identity_provider(cluster_name): - oidc_provider_issuer_url = boto3.client("eks").describe_cluster( - name=cluster_name, - )["cluster"]["identity"]["oidc"]["issuer"] - oidc_provider_issuer_endpoint = oidc_provider_issuer_url.replace("https://", "") - - account_id = boto3.client("sts").get_caller_identity()["Account"] - boto3.client("iam").delete_open_id_connect_provider( - OpenIDConnectProviderArn=f"arn:aws:iam::{account_id}:oidc-provider/{oidc_provider_issuer_endpoint}" +def _install_aws_cli_if_needed(): + """ + Ensure an AWS CLI new enough for `emr-containers create-role-associations` is installed. + + The role-association subcommands were added in AWS CLI 2.24.0. The task image may ship an + older CLI or none at all, so install a fresh one when what we find is missing or too old. + """ + log = logging.getLogger(__name__) + check = subprocess.run("aws --version", shell=True, capture_output=True, text=True, check=False) + + # `aws --version` has written to stderr on some past releases, so check both streams. + version_match = re.search(r"aws-cli/(\d+)\.(\d+)\.(\d+)", f"{check.stdout}{check.stderr}") + if check.returncode == 0 and version_match: + version = tuple(int(part) for part in version_match.groups()) + if version >= MIN_AWS_CLI_VERSION: + log.info("AWS CLI %s satisfies the minimum required version.", version) + return + log.info("AWS CLI %s is older than %s; upgrading.", version, MIN_AWS_CLI_VERSION) + else: + log.info("AWS CLI not found; installing.") + + install = subprocess.run( + """ + curl --silent "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && + unzip -q -o awscliv2.zip && + ./aws/install --update --bin-dir /usr/local/bin --install-dir /usr/local/aws-cli + """, + shell=True, + capture_output=True, + text=True, + check=False, ) + if install.returncode != 0: + raise RuntimeError(f"Failed to install the AWS CLI: {install.stderr}") @task -def update_trust_policy_execution_role(cluster_name, cluster_namespace, role_name): - # Remove any already existing trusted entities added with "update-role-trust-policy" - # Prevent getting an error "Cannot exceed quota for ACLSizePerRole" - client = boto3.client("iam") - role_trust_policy = client.get_role(RoleName=role_name)["Role"]["AssumeRolePolicyDocument"] - # We assume if the action is sts:AssumeRoleWithWebIdentity, the statement had been added with - # "update-role-trust-policy". Removing it to not exceed the quota - role_trust_policy["Statement"] = [ - statement - for statement in role_trust_policy["Statement"] - if statement["Action"] != "sts:AssumeRoleWithWebIdentity" - ] - - client.update_assume_role_policy( - RoleName=role_name, - PolicyDocument=json.dumps(role_trust_policy), +def install_pod_identity_agent(cluster_name): + """Install and wait for the EKS Pod Identity Agent add-on. + + Each run creates Pod Identity associations scoped to its own cluster. The + role's trust policy stays static. + """ + log = logging.getLogger(__name__) + eks_client = boto3.client("eks") + addon_name = "eks-pod-identity-agent" + + eks_client.create_addon(clusterName=cluster_name, addonName=addon_name) + + waiter = eks_client.get_waiter("addon_active") + waiter.wait( + clusterName=cluster_name, + addonName=addon_name, + WaiterConfig={"Delay": 10, "MaxAttempts": 60}, ) + log.info("Pod Identity Agent addon is ACTIVE on cluster %s", cluster_name) + + +@task +def create_pod_identity_role_associations(cluster_name, namespace, role_name): + # `emr-containers create-role-associations` is a CLI-only helper — it creates + # the three Pod Identity associations (client, driver, executor) that the EMR + # job pods use. There is no boto3 equivalent. + # See https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html + _install_aws_cli_if_needed() - # See https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-trust-policy.html - # The action "update-role-trust-policy" is not available in boto3, thus we need to do it using AWS CLI commands = ( - f"aws emr-containers update-role-trust-policy --cluster-name {cluster_name} " - f"--namespace {cluster_namespace} --role-name {role_name}" + f"aws emr-containers create-role-associations --cluster-name {cluster_name} " + f"--namespace {namespace} --role-name {role_name}" ) build = subprocess.Popen( @@ -178,64 +210,25 @@ def update_trust_policy_execution_role(cluster_name, cluster_namespace, role_nam raise RuntimeError(err) -@task -def wait_for_trust_policy_propagation(cluster_name, role_name): - """Validate that the IAM trust policy has propagated by checking the role's - trust policy contains the expected OIDC provider. +@task(trigger_rule=TriggerRule.ALL_DONE) +def delete_pod_identity_role_associations(cluster_name, namespace, role_name): + _install_aws_cli_if_needed() - Uses exponential backoff retries (up to 5 minutes) instead of a fixed sleep, - which avoids both wasting time when propagation is fast and failing when it's slow. - """ - log = logging.getLogger(__name__) + commands = ( + f"aws emr-containers delete-role-associations --cluster-name {cluster_name} " + f"--namespace {namespace} --role-name {role_name}" + ) - # Determine the expected OIDC provider ARN from the EKS cluster - eks_client = boto3.client("eks") - oidc_issuer_url = eks_client.describe_cluster(name=cluster_name)["cluster"]["identity"]["oidc"]["issuer"] - oidc_issuer_endpoint = oidc_issuer_url.replace("https://", "") - account_id = boto3.client("sts").get_caller_identity()["Account"] - expected_oidc_provider_arn = f"arn:aws:iam::{account_id}:oidc-provider/{oidc_issuer_endpoint}" - - @retry( - retry=retry_if_exception_type(RuntimeError), - wait=wait_exponential(multiplier=1, min=5, max=30), - stop=stop_after_delay(300), - reraise=True, + build = subprocess.Popen( + commands, + shell=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, ) - def _validate_trust_policy(): - iam_client = boto3.client("iam") - - # Verify the trust policy document contains the expected OIDC provider - role = iam_client.get_role(RoleName=role_name)["Role"] - trust_policy = role["AssumeRolePolicyDocument"] - - has_oidc_statement = False - for statement in trust_policy.get("Statement", []): - if statement.get("Action") != "sts:AssumeRoleWithWebIdentity": - continue - principal = statement.get("Principal", {}) - federated = principal.get("Federated", "") - if oidc_issuer_endpoint in federated: - has_oidc_statement = True - break - - if not has_oidc_statement: - log.info( - "Trust policy does not yet contain OIDC provider %s, retrying...", - expected_oidc_provider_arn, - ) - raise RuntimeError( - f"Trust policy for role {role_name} does not yet contain " - f"the expected OIDC provider: {expected_oidc_provider_arn}" - ) - - log.info("Trust policy document confirmed for role %s", role_name) - - _validate_trust_policy() - - # Brief buffer after IAM confirms the trust policy document — cross-service - # caches (EKS/EMR) may still serve the old policy for a few seconds. - time.sleep(15) - log.info("Trust policy validation complete, proceeding.") + _, err = build.communicate() + + if build.returncode != 0: + raise RuntimeError(err) @task(trigger_rule=TriggerRule.ALL_DONE) @@ -332,7 +325,7 @@ def delete_virtual_cluster(virtual_cluster_id): task_id="start_job", virtual_cluster_id=str(create_emr_eks_cluster.output), execution_role_arn=job_role_arn, - release_label="emr-7.0.0-latest", + release_label="emr-7.13.0-latest", job_driver=job_driver_arg, configuration_overrides=configuration_overrides_arg, name="pi.py", @@ -385,15 +378,15 @@ def delete_virtual_cluster(virtual_cluster_id): create_cluster_and_nodegroup, await_create_nodegroup, run_eksctl_commands(eks_cluster_name, eks_namespace), - update_trust_policy_execution_role(eks_cluster_name, eks_namespace, job_role_name), - wait_for_trust_policy_propagation(eks_cluster_name, job_role_name), + install_pod_identity_agent(eks_cluster_name), + create_pod_identity_role_associations(eks_cluster_name, eks_namespace, job_role_name), # TEST BODY create_emr_eks_cluster, job_starter, job_waiter, describe_pod, # TEST TEARDOWN - delete_iam_oidc_identity_provider(eks_cluster_name), + delete_pod_identity_role_associations(eks_cluster_name, eks_namespace, job_role_name), delete_virtual_cluster(str(create_emr_eks_cluster.output)), delete_eks_cluster, await_delete_eks_cluster, diff --git a/providers/amazon/tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py b/providers/amazon/tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py index ccf4a63ca4c95..732b1d5bec7f6 100644 --- a/providers/amazon/tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py +++ b/providers/amazon/tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py @@ -107,6 +107,11 @@ def glue_data_quality_recommendation_workflow(): await_evaluation_run_sensor = GlueDataQualityRuleSetEvaluationRunSensor( task_id="await_evaluation_run_sensor", evaluation_run_id=start_evaluation_run.output, + # The ruleset is generated dynamically by Glue's recommendation engine and is + # non-deterministic, so recommended rules may not always pass against the sample + # data. This system test only needs to exercise the operators/sensors end-to-end, + # not assert that every recommended rule passes, so do not fail on rule failures. + verify_result_status=False, ) chain( diff --git a/providers/amazon/tests/unit/amazon/aws/secrets/test_secrets_manager.py b/providers/amazon/tests/unit/amazon/aws/secrets/test_secrets_manager.py index ab345a9f38d84..3967405d649f7 100644 --- a/providers/amazon/tests/unit/amazon/aws/secrets/test_secrets_manager.py +++ b/providers/amazon/tests/unit/amazon/aws/secrets/test_secrets_manager.py @@ -24,6 +24,10 @@ from airflow.providers.amazon.aws.secrets.secrets_manager import TEAM_SEP, SecretsManagerBackend +from tests_common.test_utils.config import conf_vars + +multi_team_enabled = conf_vars({("core", "multi_team"): "True"}) + class TestSecretsManagerBackend: @mock.patch("airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend.get_conn_value") @@ -80,6 +84,7 @@ def test_get_conn_value_with_team_name(self): returned_uri = secrets_manager_backend.get_conn_value(conn_id="test_postgres", team_name="my_team") assert returned_uri == "postgresql://airflow:airflow@host:5432/airflow" + @multi_team_enabled @mock_aws def test_global_caller_cannot_access_team_scoped_connection(self): secret_id = "airflow/connections/my_team--test_postgres" @@ -93,6 +98,7 @@ def test_global_caller_cannot_access_team_scoped_connection(self): assert secrets_manager_backend.get_conn_value(conn_id="my_team--test_postgres") is None + @multi_team_enabled @mock_aws def test_another_teams_secret_is_not_reachable(self): """A caller scoped to one team must not reach another team's secret by naming it.""" @@ -103,6 +109,7 @@ def test_another_teams_secret_is_not_reachable(self): assert backend.get_conn_value(conn_id="my_team--test_postgres", team_name="other_team") is None + @multi_team_enabled @mock_aws def test_team_whose_name_extends_the_callers_is_not_reachable(self): """A prefix match on the caller's own namespace is not proof of ownership.""" @@ -113,6 +120,7 @@ def test_team_whose_name_extends_the_callers_is_not_reachable(self): assert backend.get_conn_value(conn_id="my_team--prod--test_postgres", team_name="my_team") is None + @multi_team_enabled @mock_aws def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace(self): """The team scoped name is not safe by construction -- the id can extend it. @@ -129,6 +137,7 @@ def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace(self): assert backend.get_conn_value(conn_id="prod--test_postgres", team_name="my_team") is None + @multi_team_enabled @mock_aws def test_refusing_an_ambiguous_id_is_logged(self, caplog): """A silent ``None`` is indistinguishable from a missing secret, so the refusal is logged. @@ -158,6 +167,22 @@ def test_refusing_an_ambiguous_id_is_logged(self, caplog): assert sum(refused_id in r.getMessage() for r in refusals) == 1 assert all(TEAM_SEP in r.getMessage() for r in refusals) + @mock_aws + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self): + """No team scoped secret can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + secret_id = "airflow/connections/prod--test_postgres" + create_param = { + "Name": secret_id, + "SecretString": "postgresql://airflow:airflow@host:5432/airflow", + } + backend = SecretsManagerBackend() + backend.client.create_secret(**create_param) + + assert backend.get_conn_value(conn_id="prod--test_postgres") == ( + "postgresql://airflow:airflow@host:5432/airflow" + ) + @mock_aws def test_team_caller_falls_back_to_global_connection(self): secret_id = "airflow/connections/test_postgres" @@ -209,6 +234,7 @@ def test_get_variable_with_team_name(self): assert secrets_manager_backend.get_variable(key="hello", team_name="my_team") == "world" + @multi_team_enabled @mock_aws def test_global_caller_cannot_access_team_scoped_variable(self): secret_id = "airflow/variables/my_team--hello" diff --git a/providers/amazon/tests/unit/amazon/aws/secrets/test_systems_manager.py b/providers/amazon/tests/unit/amazon/aws/secrets/test_systems_manager.py index 09734819e13f1..8795eeb53ea54 100644 --- a/providers/amazon/tests/unit/amazon/aws/secrets/test_systems_manager.py +++ b/providers/amazon/tests/unit/amazon/aws/secrets/test_systems_manager.py @@ -31,6 +31,8 @@ from tests_common.test_utils.config import conf_vars +multi_team_enabled = conf_vars({("core", "multi_team"): "True"}) + URI_CONNECTION = pytest.param( "postgres://my-login:my-pass@my-host:5432/my-schema?param1=val1¶m2=val2", id="uri-connection" ) @@ -116,6 +118,7 @@ def test_get_conn_value_with_team_name(self): returned_uri = ssm_backend.get_conn_value(conn_id="test_postgres", team_name="my_team") assert returned_uri == "postgresql://airflow:airflow@host:5432/airflow" + @multi_team_enabled @mock_aws def test_global_caller_cannot_access_team_scoped_connection(self): param = { @@ -127,6 +130,7 @@ def test_global_caller_cannot_access_team_scoped_connection(self): ssm_backend.client.put_parameter(**param) assert ssm_backend.get_conn_value(conn_id="my_team--test_postgres") is None + @multi_team_enabled @mock_aws def test_another_teams_secret_is_not_reachable(self): """A caller scoped to one team must not reach another team's parameter by naming it.""" @@ -140,6 +144,7 @@ def test_another_teams_secret_is_not_reachable(self): assert ssm_backend.get_conn_value(conn_id="my_team--test_postgres", team_name="other_team") is None + @multi_team_enabled @mock_aws def test_team_whose_name_extends_the_callers_is_not_reachable(self): """A prefix match on the caller's own namespace is not proof of ownership.""" @@ -153,6 +158,7 @@ def test_team_whose_name_extends_the_callers_is_not_reachable(self): assert ssm_backend.get_conn_value(conn_id="my_team--prod--test_postgres", team_name="my_team") is None + @multi_team_enabled @mock_aws def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace(self): """The team scoped name is not safe by construction -- the id can extend it. @@ -172,6 +178,7 @@ def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace(self): assert ssm_backend.get_conn_value(conn_id="prod--test_postgres", team_name="my_team") is None + @multi_team_enabled @mock_aws def test_refusing_an_ambiguous_id_is_logged(self, caplog): """A silent ``None`` is indistinguishable from a missing secret, so the refusal is logged. @@ -201,6 +208,22 @@ def test_refusing_an_ambiguous_id_is_logged(self, caplog): assert sum(refused_id in r.getMessage() for r in refusals) == 1 assert all(TEAM_SEP in r.getMessage() for r in refusals) + @mock_aws + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self): + """No team scoped parameter can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + param = { + "Name": "/airflow/connections/prod--test_postgres", + "Type": "String", + "Value": "postgresql://airflow:airflow@host:5432/airflow", + } + ssm_backend = SystemsManagerParameterStoreBackend() + ssm_backend.client.put_parameter(**param) + + assert ssm_backend.get_conn_value(conn_id="prod--test_postgres") == ( + "postgresql://airflow:airflow@host:5432/airflow" + ) + @mock_aws def test_team_caller_falls_back_to_global_connection(self): param = { @@ -267,6 +290,7 @@ def test_get_variable_with_team_name(self): assert ssm_backend.get_variable(key="hello", team_name="my_team") == "world" + @multi_team_enabled @mock_aws def test_global_caller_cannot_access_team_scoped_variable(self): param = {"Name": "/airflow/variables/my_team--hello", "Type": "String", "Value": "world"} diff --git a/providers/amazon/tests/unit/amazon/aws/sensors/test_bedrock.py b/providers/amazon/tests/unit/amazon/aws/sensors/test_bedrock.py index 559926579a453..e3e3792b29095 100644 --- a/providers/amazon/tests/unit/amazon/aws/sensors/test_bedrock.py +++ b/providers/amazon/tests/unit/amazon/aws/sensors/test_bedrock.py @@ -239,12 +239,24 @@ def test_poke_intermediate_states(self, mock_conn, state): assert self.sensor.poke({}) is False @pytest.mark.parametrize("state", SENSOR.FAILURE_STATES) + @pytest.mark.parametrize( + ("job_detail", "expected_suffix"), + [ + pytest.param({}, "", id="no reasons"), + pytest.param( + {"failureReasons": ["User is not authorized", "index not found"]}, + " Failure reasons: User is not authorized; index not found", + id="with reasons", + ), + ], + ) @mock.patch.object(BedrockAgentHook, "conn") - def test_poke_failure_states(self, mock_conn, state): - mock_conn.get_ingestion_job.return_value = {"ingestionJob": {"status": state}} + def test_poke_failure_states(self, mock_conn, state, job_detail, expected_suffix): + mock_conn.get_ingestion_job.return_value = {"ingestionJob": {"status": state, **job_detail}} sensor = self.SENSOR(**self.default_op_kwargs, aws_conn_id=None) - with pytest.raises(AirflowException, match=sensor.FAILURE_MESSAGE): + with pytest.raises(AirflowException, match=sensor.FAILURE_MESSAGE) as exception_info: sensor.poke({}) + assert str(exception_info.value) == f"{sensor.FAILURE_MESSAGE}{expected_suffix}" class TestBedrockBatchInferenceSensor: diff --git a/providers/amazon/tests/unit/amazon/aws/transfers/test_sql_to_s3.py b/providers/amazon/tests/unit/amazon/aws/transfers/test_sql_to_s3.py index 5fb50fff9fe60..3a960d1fd97d9 100644 --- a/providers/amazon/tests/unit/amazon/aws/transfers/test_sql_to_s3.py +++ b/providers/amazon/tests/unit/amazon/aws/transfers/test_sql_to_s3.py @@ -24,12 +24,15 @@ import pandas as pd import polars as pl import pytest +from packaging.version import Version from airflow.exceptions import AirflowProviderDeprecationWarning from airflow.models import Connection from airflow.providers.amazon.aws.transfers.sql_to_s3 import SqlToS3Operator from airflow.providers.common.compat.sdk import AirflowException +PANDAS_3_PLUS = Version(pd.__version__).major >= 3 + class TestSqlToS3Operator: @pytest.mark.parametrize( @@ -156,7 +159,12 @@ def test_fix_dtypes(self, params): ) dirty_df = pd.DataFrame({"strings": ["a", "b", None], "ints": [1, 2, None]}) op._fix_dtypes(df=dirty_df, file_format=op.file_format) - assert dirty_df["strings"].values[2] == params["null_string_result"] + if PANDAS_3_PLUS: + # pandas 3 infers a str column rather than object, and keeps its missing values as NA + # instead of the object None (csv) or the "None" it used to be stringified to (parquet) + assert pd.isna(dirty_df["strings"].values[2]) + else: + assert dirty_df["strings"].values[2] == params["null_string_result"] assert dirty_df["ints"].dtype.kind == "i" @mock.patch("airflow.providers.amazon.aws.transfers.sql_to_s3.S3Hook") diff --git a/providers/apache/beam/docs/changelog.rst b/providers/apache/beam/docs/changelog.rst index dcac0138bc28c..693b13c8fb5e9 100644 --- a/providers/apache/beam/docs/changelog.rst +++ b/providers/apache/beam/docs/changelog.rst @@ -268,7 +268,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 6.0.1 ..... diff --git a/providers/apache/hdfs/README.rst b/providers/apache/hdfs/README.rst index f965b90f53b6c..500ea51e28cf2 100644 --- a/providers/apache/hdfs/README.rst +++ b/providers/apache/hdfs/README.rst @@ -51,19 +51,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"`` ``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"`` ``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"`` ``fastavro`` ``>=1.12.1; python_version >= "3.14"`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================== The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/apache/hdfs/docs/changelog.rst b/providers/apache/hdfs/docs/changelog.rst index fc58389b7859f..7853db98895f2 100644 --- a/providers/apache/hdfs/docs/changelog.rst +++ b/providers/apache/hdfs/docs/changelog.rst @@ -33,11 +33,12 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 4.12.1 ...... diff --git a/providers/apache/hdfs/docs/index.rst b/providers/apache/hdfs/docs/index.rst index 8333ad6255c8d..d0bf84cd7407d 100644 --- a/providers/apache/hdfs/docs/index.rst +++ b/providers/apache/hdfs/docs/index.rst @@ -85,19 +85,19 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``hdfs[avro,dataframe,kerberos]`` ``>=2.5.4; python_version < "3.12"`` ``hdfs[avro,dataframe,kerberos]`` ``>=2.7.3; python_version >= "3.12"`` ``fastavro`` ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"`` ``fastavro`` ``>=1.12.1; python_version >= "3.14"`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================== Downloading official packages ----------------------------- diff --git a/providers/apache/hdfs/provider.yaml b/providers/apache/hdfs/provider.yaml index 155137037afa3..d6f92d7013778 100644 --- a/providers/apache/hdfs/provider.yaml +++ b/providers/apache/hdfs/provider.yaml @@ -117,6 +117,10 @@ hooks: logging: - airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsTaskHandler +remote-logging: + - classpath: airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsRemoteLogIO + scheme: hdfs + connection-types: - hook-class-name: airflow.providers.apache.hdfs.hooks.webhdfs.WebHDFSHook hook-name: "Apache WebHDFS" diff --git a/providers/apache/hdfs/pyproject.toml b/providers/apache/hdfs/pyproject.toml index 398eedb8f7099..d309ff8fed80c 100644 --- a/providers/apache/hdfs/pyproject.toml +++ b/providers/apache/hdfs/pyproject.toml @@ -65,18 +65,9 @@ dependencies = [ 'hdfs[avro,dataframe,kerberos]>=2.7.3;python_version>="3.12"', 'fastavro>=1.10.0; python_version>="3.13" and python_version<"3.14"', 'fastavro>=1.12.1; python_version>="3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] [dependency-groups] diff --git a/providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py b/providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py index 381cc777b4547..98a0936e42645 100644 --- a/providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py +++ b/providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py @@ -66,6 +66,12 @@ def get_provider_info(): {"integration-name": "WebHDFS", "python-modules": ["airflow.providers.apache.hdfs.hooks.webhdfs"]} ], "logging": ["airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsTaskHandler"], + "remote-logging": [ + { + "classpath": "airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsRemoteLogIO", + "scheme": "hdfs", + } + ], "connection-types": [ { "hook-class-name": "airflow.providers.apache.hdfs.hooks.webhdfs.WebHDFSHook", diff --git a/providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py b/providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py index 22304637db237..5694c62602ae3 100644 --- a/providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py +++ b/providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py @@ -17,6 +17,7 @@ # under the License. from __future__ import annotations +import inspect import logging import os import shutil @@ -46,6 +47,31 @@ class HdfsRemoteLogIO(LoggingMixin): # noqa: D101 processors = () + @classmethod + def from_config(cls) -> HdfsRemoteLogIO: + """Build the remote log IO from Airflow logging configuration.""" + remote_task_handler_kwargs = conf.getjson("logging", "remote_task_handler_kwargs", fallback={}) + if not isinstance(remote_task_handler_kwargs, dict): + raise ValueError( + "logging/remote_task_handler_kwargs must be a JSON object (a python dict), we got " + f"{type(remote_task_handler_kwargs)}" + ) + # remote_task_handler_kwargs mixes FileTaskHandler kwargs with IO kwargs; only the + # latter belong to this class (same split as airflow_local_settings.py). + fth_params = frozenset(inspect.signature(FileTaskHandler.__init__).parameters) - { + "self", + "base_log_folder", + } + io_kwargs = {k: v for k, v in remote_task_handler_kwargs.items() if k not in fth_params} + return cls( + **{ + "base_log_folder": os.path.expanduser(conf.get_mandatory_value("logging", "base_log_folder")), + "remote_base": urlsplit(conf.get_mandatory_value("logging", "remote_base_log_folder")).path, + "delete_local_copy": conf.getboolean("logging", "delete_local_logs"), + } + | io_kwargs, + ) + def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None: """Upload the given log path to the remote storage.""" path = Path(path) diff --git a/providers/apache/hdfs/tests/unit/apache/hdfs/log/test_hdfs_task_handler.py b/providers/apache/hdfs/tests/unit/apache/hdfs/log/test_hdfs_task_handler.py index b5e2adb757797..bd82513cbff47 100644 --- a/providers/apache/hdfs/tests/unit/apache/hdfs/log/test_hdfs_task_handler.py +++ b/providers/apache/hdfs/tests/unit/apache/hdfs/log/test_hdfs_task_handler.py @@ -25,7 +25,7 @@ import pytest from airflow.providers.apache.hdfs.hooks.webhdfs import WebHDFSHook -from airflow.providers.apache.hdfs.log.hdfs_task_handler import HdfsTaskHandler +from airflow.providers.apache.hdfs.log.hdfs_task_handler import HdfsRemoteLogIO, HdfsTaskHandler from airflow.utils.state import TaskInstanceState from airflow.utils.timezone import datetime @@ -37,6 +37,86 @@ DEFAULT_DATE = datetime(2020, 8, 10) +class TestHdfsRemoteLogIOFromConfig: + @conf_vars( + { + ("logging", "base_log_folder"): "~/airflow/logs", + ("logging", "remote_base_log_folder"): "hdfs://namenode/remote/log/location", + ("logging", "delete_local_logs"): "True", + } + ) + def test_from_config(self): + subject = HdfsRemoteLogIO.from_config() + + assert subject.remote_base == "/remote/log/location" + assert subject.base_log_folder == Path(os.path.expanduser("~/airflow/logs")) + assert subject.delete_local_copy is True + + @conf_vars( + { + ("logging", "base_log_folder"): "/tmp/airflow/logs", + ("logging", "remote_base_log_folder"): "hdfs://namenode/remote/log/location", + ("logging", "delete_local_logs"): "False", + ("logging", "remote_task_handler_kwargs"): '{"delete_local_copy": true, "max_bytes": 1024}', + } + ) + def test_from_config_applies_io_kwargs_and_filters_file_handler_kwargs(self): + subject = HdfsRemoteLogIO.from_config() + + assert subject.delete_local_copy is True + assert not hasattr(subject, "max_bytes") + + @conf_vars({("logging", "remote_task_handler_kwargs"): '["not", "a", "dict"]'}) + def test_from_config_rejects_non_dict_remote_task_handler_kwargs(self): + with pytest.raises(ValueError, match="remote_task_handler_kwargs"): + HdfsRemoteLogIO.from_config() + + def test_provider_registers_hdfs_scheme(self): + from airflow.providers_manager import ProvidersManager + + manager = ProvidersManager() + if not hasattr(manager, "remote_logging_handler_by_scheme"): + pytest.skip("Airflow core does not support remote logging provider dispatch") + + info = manager.remote_logging_handler_by_scheme("hdfs") + + assert info is not None + assert info.classpath == "airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsRemoteLogIO" + + @pytest.mark.parametrize( + "manager_classpath", + [ + pytest.param("airflow.providers_manager.ProvidersManager", id="core"), + pytest.param( + "airflow.sdk.providers_manager_runtime.ProvidersManagerTaskRuntime", id="task-runtime" + ), + ], + ) + @conf_vars( + { + ("logging", "remote_logging"): "True", + ("logging", "remote_base_log_folder"): "hdfs://namenode/remote/log/location", + ("logging", "remote_log_conn_id"): "webhdfs_default", + } + ) + def test_resolve_remote_task_log_uses_provider_dispatch_not_local_settings(self, manager_classpath): + factory = pytest.importorskip("airflow._shared.logging.factory") + from airflow._shared.module_loading import import_string + from airflow.configuration import conf + + with mock.patch.object(factory, "discover_remote_log_handler", autospec=True) as legacy_discover: + remote_task_log, conn_id = factory.resolve_remote_task_log( + conf=conf, + providers_manager=import_string(manager_classpath)(), + import_string=import_string, + ) + + assert isinstance(remote_task_log, HdfsRemoteLogIO) + assert remote_task_log.remote_base == "/remote/log/location" + assert conn_id == "webhdfs_default" + legacy_discover.assert_not_called() + + class TestHdfsTaskHandler: @pytest.fixture(autouse=True) def ti(self, create_task_instance, create_log_template): diff --git a/providers/apache/hive/README.rst b/providers/apache/hive/README.rst index 1f20df918edf5..e7e89ad52f7da 100644 --- a/providers/apache/hive/README.rst +++ b/providers/apache/hive/README.rst @@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``hmsclient`` ``>=0.1.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyhive[hive_pure_sasl]`` ``>=0.7.0`` ``jmespath`` ``>=0.7.0`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/apache/hive/docs/changelog.rst b/providers/apache/hive/docs/changelog.rst index fb2b9a6c02a7d..58ca935435278 100644 --- a/providers/apache/hive/docs/changelog.rst +++ b/providers/apache/hive/docs/changelog.rst @@ -38,11 +38,12 @@ Bug Fixes Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): * ``Use common.compat.sdk for timezone imports in providers (#70492)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 9.6.0 ..... @@ -392,7 +393,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 9.0.1 ..... diff --git a/providers/apache/hive/docs/index.rst b/providers/apache/hive/docs/index.rst index b19c2909172b8..8008c3b8acc5f 100644 --- a/providers/apache/hive/docs/index.rst +++ b/providers/apache/hive/docs/index.rst @@ -99,19 +99,19 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``hmsclient`` ``>=0.1.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyhive[hive_pure_sasl]`` ``>=0.7.0`` ``jmespath`` ``>=0.7.0`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/apache/hive/pyproject.toml b/providers/apache/hive/pyproject.toml index 8565a499001ef..c7b62d833bb36 100644 --- a/providers/apache/hive/pyproject.toml +++ b/providers/apache/hive/pyproject.toml @@ -63,18 +63,9 @@ dependencies = [ "apache-airflow-providers-common-compat>=1.12.0", "apache-airflow-providers-common-sql>=1.32.0", "hmsclient>=0.1.0", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "pyhive[hive_pure_sasl]>=0.7.0", "jmespath>=0.7.0", ] diff --git a/providers/apache/livy/docs/changelog.rst b/providers/apache/livy/docs/changelog.rst index fed32f6c58ede..76c2c2a56baf6 100644 --- a/providers/apache/livy/docs/changelog.rst +++ b/providers/apache/livy/docs/changelog.rst @@ -315,7 +315,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.1.0 diff --git a/providers/apache/spark/docs/changelog.rst b/providers/apache/spark/docs/changelog.rst index cb858c4556947..4aef1f7cfba6a 100644 --- a/providers/apache/spark/docs/changelog.rst +++ b/providers/apache/spark/docs/changelog.rst @@ -35,6 +35,7 @@ Bug Fixes ~~~~~~~~~ * ``Anchor spark-submit log trail on actual exception instead of lines (#70513)`` +* ``Spark: Fix missing space in YARN waitAppCompletion error message (#71055)`` Misc ~~~~ diff --git a/providers/apache/spark/pyproject.toml b/providers/apache/spark/pyproject.toml index b7209a2f35cba..37e6f6dd89802 100644 --- a/providers/apache/spark/pyproject.toml +++ b/providers/apache/spark/pyproject.toml @@ -89,7 +89,7 @@ dev = [ "apache-airflow-providers-common-compat", # Additional devel dependencies (do not remove this line and add extra development dependencies) "apache-airflow-providers-openlineage", - "pyspark" + "pyspark>=4.0.0" ] # To build docs: diff --git a/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py b/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py index 096f9ca84fc37..d70729cb27f21 100644 --- a/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py +++ b/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py @@ -152,7 +152,7 @@ class _YarnSparkSubmitBackend(_SparkSubmitDeploymentBackend): def submit_job(self, context: Context) -> str | None: if self.hook._conf.get("spark.yarn.submit.waitAppCompletion", "").strip().lower() == "true": raise ValueError( - "spark.yarn.submit.waitAppCompletion=true cannot be set for cluster mode as it conflicts" + "spark.yarn.submit.waitAppCompletion=true cannot be set for cluster mode as it conflicts " "with the need to exit spark-submit immediately to persist the application ID for tracking. " "Either remove the explicit conf or set durable=False." ) diff --git a/providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py b/providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py index 18a3928e7910c..953af3408d051 100644 --- a/providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py +++ b/providers/apache/spark/tests/unit/apache/spark/operators/test_spark_submit.py @@ -819,7 +819,10 @@ def test_yarn_raises_if_wait_app_completion_true(self): hook._conf = {"spark.yarn.submit.waitAppCompletion": "true"} operator._hook = hook - with pytest.raises(ValueError, match="waitAppCompletion=true"): + with pytest.raises( + ValueError, + match=r"spark\.yarn\.submit\.waitAppCompletion=true cannot be set for cluster mode as it conflicts with the need", + ): operator.submit_job(context={}) def test_yarn_poll_tolerates_transient_resourcemanager_failures(self): diff --git a/providers/arangodb/docs/changelog.rst b/providers/arangodb/docs/changelog.rst index d3826f3ba3f62..2e2f18d47cbda 100644 --- a/providers/arangodb/docs/changelog.rst +++ b/providers/arangodb/docs/changelog.rst @@ -242,7 +242,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 2.7.1 ..... diff --git a/providers/asana/docs/changelog.rst b/providers/asana/docs/changelog.rst index 928e84c55cf72..b4acfba25e485 100644 --- a/providers/asana/docs/changelog.rst +++ b/providers/asana/docs/changelog.rst @@ -209,7 +209,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 2.8.0 ..... diff --git a/providers/celery/docs/changelog.rst b/providers/celery/docs/changelog.rst index e1de5e1801dc8..ba247882bb29a 100644 --- a/providers/celery/docs/changelog.rst +++ b/providers/celery/docs/changelog.rst @@ -638,7 +638,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.10.1 ...... diff --git a/providers/clickhousedb/README.rst b/providers/clickhousedb/README.rst index 0c4f4d8cf1b8f..e7209c804916f 100644 --- a/providers/clickhousedb/README.rst +++ b/providers/clickhousedb/README.rst @@ -23,7 +23,7 @@ Package ``apache-airflow-providers-clickhousedb`` -Release: ``1.0.1`` +Release: ``1.0.0`` `ClickHouse `__ @@ -36,7 +36,7 @@ This is a provider package for ``clickhousedb`` provider. All classes for this p are in ``airflow.providers.clickhousedb`` python package. You can find package information and changelog for the provider -in the `documentation `_. +in the `documentation `_. Installation ------------ @@ -59,4 +59,4 @@ PIP package Version required ======================================= ================== The changelog for the provider package can be found in the -`changelog `_. +`changelog `_. diff --git a/providers/clickhousedb/docs/changelog.rst b/providers/clickhousedb/docs/changelog.rst index 970500fdf2e07..21aaf99532cac 100644 --- a/providers/clickhousedb/docs/changelog.rst +++ b/providers/clickhousedb/docs/changelog.rst @@ -27,21 +27,6 @@ Changelog --------- -1.0.1 -..... - -Misc -~~~~ - -* ``Change Clickhouse lifecycle to incubation (#69166)`` - -.. Below changes are excluded from the changelog. Move them to - appropriate section above if needed. Do not delete the lines(!): - * ``Fix ClickHouse test_connection unit test failing with newer clickhouse-connect (#70114)`` - * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` - * ``Prepare providers release 2026-07-22 (#70256)`` - - 1.0.0 ..... diff --git a/providers/clickhousedb/docs/index.rst b/providers/clickhousedb/docs/index.rst index b399de766ac96..8061b05cb0f00 100644 --- a/providers/clickhousedb/docs/index.rst +++ b/providers/clickhousedb/docs/index.rst @@ -77,7 +77,7 @@ apache-airflow-providers-clickhousedb package `ClickHouse `__ -Release: 1.0.1 +Release: 1.0.0 Provider package ---------------- @@ -111,5 +111,5 @@ Downloading official packages You can download officially released packages and verify their checksums and signatures from the `Official Apache Download site `_ -* `The apache-airflow-providers-clickhousedb 1.0.1 sdist package `_ (`asc `__, `sha512 `__) -* `The apache-airflow-providers-clickhousedb 1.0.1 wheel package `_ (`asc `__, `sha512 `__) +* `The apache-airflow-providers-clickhousedb 1.0.0 sdist package `_ (`asc `__, `sha512 `__) +* `The apache-airflow-providers-clickhousedb 1.0.0 wheel package `_ (`asc `__, `sha512 `__) diff --git a/providers/clickhousedb/provider.yaml b/providers/clickhousedb/provider.yaml index 0555f1029eed1..43fdb67daeae0 100644 --- a/providers/clickhousedb/provider.yaml +++ b/providers/clickhousedb/provider.yaml @@ -29,7 +29,6 @@ source-date-epoch: 1785633505 # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have # to be done in the same PR versions: - - 1.0.1 - 1.0.0 integrations: diff --git a/providers/clickhousedb/pyproject.toml b/providers/clickhousedb/pyproject.toml index ccc1b6c3b4de9..dc19451b828f7 100644 --- a/providers/clickhousedb/pyproject.toml +++ b/providers/clickhousedb/pyproject.toml @@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi" [project] name = "apache-airflow-providers-clickhousedb" -version = "1.0.1" +version = "1.0.0" description = "Provider package apache-airflow-providers-clickhousedb for Apache Airflow" readme = "README.rst" license = "Apache-2.0" @@ -100,8 +100,8 @@ apache-airflow-providers-common-sql = {workspace = true} apache-airflow-providers-standard = {workspace = true} [project.urls] -"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-clickhousedb/1.0.1" -"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-clickhousedb/1.0.1/changelog.html" +"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-clickhousedb/1.0.0" +"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-clickhousedb/1.0.0/changelog.html" "Bug Tracker" = "https://github.com/apache/airflow/issues" "Source Code" = "https://github.com/apache/airflow" "Slack Chat" = "https://s.apache.org/airflow-slack" diff --git a/providers/clickhousedb/src/airflow/providers/clickhousedb/__init__.py b/providers/clickhousedb/src/airflow/providers/clickhousedb/__init__.py index 4e0e1fe0d6807..112a7774d4227 100644 --- a/providers/clickhousedb/src/airflow/providers/clickhousedb/__init__.py +++ b/providers/clickhousedb/src/airflow/providers/clickhousedb/__init__.py @@ -29,7 +29,7 @@ __all__ = ["__version__"] -__version__ = "1.0.1" +__version__ = "1.0.0" if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse( "2.11.0" diff --git a/providers/cncf/kubernetes/docs/changelog.rst b/providers/cncf/kubernetes/docs/changelog.rst index 8f8975bc91759..6530ffdddc38c 100644 --- a/providers/cncf/kubernetes/docs/changelog.rst +++ b/providers/cncf/kubernetes/docs/changelog.rst @@ -64,6 +64,7 @@ Doc-only .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): * ``Use common.compat.sdk for timezone imports in providers (#70492)`` + * ``Fix version_added for the concurrent pod creation options (#71157)`` 10.20.0 ....... @@ -936,7 +937,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 10.2.0 ...... diff --git a/providers/cncf/kubernetes/provider.yaml b/providers/cncf/kubernetes/provider.yaml index cfecb10cba9b7..895e9739006c3 100644 --- a/providers/cncf/kubernetes/provider.yaml +++ b/providers/cncf/kubernetes/provider.yaml @@ -375,7 +375,7 @@ config: per-call latency is high (network round-trip, admission webhooks). Pod templates are still built synchronously; only the create API calls are parallelized. - version_added: 10.20.0 + version_added: 10.21.0 type: boolean example: ~ default: "False" @@ -386,7 +386,7 @@ config: server (and admission webhooks) to avoid tripping API priority-and-fairness or rate limits. Has no effect when ``async_pod_creation`` is False. When set to 0 the limit falls back to ``worker_pods_creation_batch_size``. - version_added: 10.20.0 + version_added: 10.21.0 type: integer example: "16" default: "0" diff --git a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py index 8a2daec956f4e..5086168855083 100644 --- a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py +++ b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py @@ -224,14 +224,14 @@ def get_provider_info(): }, "async_pod_creation": { "description": "Create worker pods concurrently within each scheduler loop instead of\nsequentially. When enabled, the ``worker_pods_creation_batch_size`` pods\ndequeued per loop are submitted to the Kubernetes API concurrently (bounded\nby ``pod_creation_max_concurrency``) using the asynchronous Kubernetes client.\nThis reduces the time the scheduler loop spends blocked on pod creation when\nper-call latency is high (network round-trip, admission webhooks). Pod\ntemplates are still built synchronously; only the create API calls are\nparallelized.\n", - "version_added": "10.20.0", + "version_added": "10.21.0", "type": "boolean", "example": None, "default": "False", }, "pod_creation_max_concurrency": { "description": "Maximum number of concurrent pod-creation API calls when ``async_pod_creation``\nis enabled. Bounds the burst of simultaneous requests to the Kubernetes API\nserver (and admission webhooks) to avoid tripping API priority-and-fairness or\nrate limits. Has no effect when ``async_pod_creation`` is False. When set to 0\nthe limit falls back to ``worker_pods_creation_batch_size``.\n", - "version_added": "10.20.0", + "version_added": "10.21.0", "type": "integer", "example": "16", "default": "0", diff --git a/providers/cohere/docs/changelog.rst b/providers/cohere/docs/changelog.rst index 2a01b1ca614e3..e5faf279421bd 100644 --- a/providers/cohere/docs/changelog.rst +++ b/providers/cohere/docs/changelog.rst @@ -243,7 +243,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 1.4.1 ..... diff --git a/providers/common/ai/docs/changelog.rst b/providers/common/ai/docs/changelog.rst index a39121f8ea6f6..06f97b63bd211 100644 --- a/providers/common/ai/docs/changelog.rst +++ b/providers/common/ai/docs/changelog.rst @@ -25,6 +25,18 @@ Changelog --------- +.. note:: + The ``query`` tool of ``SQLToolset`` and ``DataFusionToolset`` returns a different + shape. Rows were a dict per row alongside a ``count`` of every matching row: + ``{"rows": [{"id": 1, "name": "a"}], "count": 900}``. They are now columnar, and + ``row_count`` counts the rows actually returned: + ``{"columns": ["id", "name"], "rows": [[1, "a"]], "row_count": 1}``. A truncated + result also carries ``truncated_by`` -- ``max_rows`` or the new ``max_result_bytes`` + -- and ``total_rows`` appears only when the driver reports a trustworthy query + total. The tool's own description states the new shape, so agents adapt without + changes; update any system prompt that describes the old shape, and any code + calling ``toolset.call_tool("query", ...)`` directly. + 0.7.0 ..... @@ -52,6 +64,8 @@ Features * ``Use task state store for 'common.ai' durable execution on Airflow 3.3+ (#68926)`` * ``Support '.txt' files in 'LLMFileAnalysisOperator' (#70431)`` * ``Support cloud URI globs in DocumentLoaderOperator (#70299)`` +* ``Support require_approval in LLMBranchOperator (#70651)`` +* ``Render multi-branch review choices as a multi-select (#71046)`` Bug Fixes ~~~~~~~~~ diff --git a/providers/common/ai/docs/operators/llm_branch.rst b/providers/common/ai/docs/operators/llm_branch.rst index 42db970427a6d..21558b2277f02 100644 --- a/providers/common/ai/docs/operators/llm_branch.rst +++ b/providers/common/ai/docs/operators/llm_branch.rst @@ -75,6 +75,34 @@ With multiple branches: :start-after: [START howto_decorator_llm_branch_multi] :end-before: [END howto_decorator_llm_branch_multi] +Human-in-the-Loop Approval +-------------------------- + +Set ``require_approval=True`` to pause the task after the LLM chooses the +branch(es) and wait for a human reviewer to approve the choice before any +downstream task is skipped. The review form shows the LLM's choice and the +valid downstream task IDs. When ``allow_modifications=True``, the reviewer +can also change the choice — rendered as a dropdown of the downstream task +IDs, or a multi-select of them with ``allow_multiple_branches=True``. The +reviewed branch(es) are validated +against the downstream task IDs before branching: + +.. exampleinclude:: /../../ai/src/airflow/providers/common/ai/example_dags/example_llm_branch.py + :language: python + :start-after: [START howto_operator_llm_branch_approval] + :end-before: [END howto_operator_llm_branch_approval] + +Rejecting the review, or letting ``approval_timeout`` expire, **fails** the +task (``HITLRejectException`` / ``HITLTimeoutError``), so downstream tasks +end up ``upstream_failed`` rather than skipped. + +``require_approval=True`` requires a string prompt: a decorated callable +returning a ``Sequence[UserContent]`` raises ``TypeError`` before the LLM +call. + +``approval_timeout`` and the rest of the approval behaviour are inherited +from :ref:`LLMOperator `. + How It Works ------------ @@ -99,6 +127,12 @@ Parameters task ID. When ``True`` the LLM may return one or more task IDs. - ``agent_params``: Additional keyword arguments passed to the pydantic-ai ``Agent`` constructor (e.g. ``retries``, ``model_settings``). Supports Jinja templating. +- ``require_approval``: If ``True``, the task pauses after the LLM chooses the + branch(es) and waits for human review before branching. Default ``False``. +- ``approval_timeout``: Maximum time to wait for a review (``timedelta``). ``None`` + means wait indefinitely. Default ``None``. +- ``allow_modifications``: If ``True``, the reviewer can change the chosen + branch(es) before approving. Default ``False``. Logging ------- diff --git a/providers/common/ai/docs/retry_policies.rst b/providers/common/ai/docs/retry_policies.rst index b498050b6851a..c1962e41ce869 100644 --- a/providers/common/ai/docs/retry_policies.rst +++ b/providers/common/ai/docs/retry_policies.rst @@ -71,8 +71,9 @@ How it works When a task fails, ``LLMRetryPolicy``: 1. Sends the exception message to the configured LLM. By default, the message - is first masked through Airflow's secrets masker (see ``redact_exception`` - below) before it is added to the prompt. + is first masked through Airflow's secrets masker (see ``redactor`` below) + and truncated to ``max_exception_length`` characters before it is added + to the prompt. 2. The LLM classifies the error into a category (``rate_limit``, ``auth``, ``network``, ``data``, ``transient``, ``permanent``) 3. Based on the classification, returns RETRY (with a suggested delay) or FAIL @@ -158,22 +159,71 @@ Parameters * - ``timeout`` - 30.0 - Max seconds to wait for the LLM response before falling back. + * - ``redactor`` + - None (uses ``redact_registered_secrets``) + - Callable ``(str) -> str`` applied to the exception's string + representation before it is added to the classification prompt. The + default only masks values already registered via ``mask_secret()`` + (e.g. connection passwords Airflow captured while resolving the + failing task's connections) -- it is not general-purpose PII + detection and will not catch arbitrary sensitive strings that were + never registered as secrets. Passing a custom callable **replaces** + the default masker entirely rather than stacking on top of it. * - ``redact_exception`` - True - - When ``True``, the exception's string representation is passed through - Airflow's secrets masker before being added to the classification - prompt. This only masks values already registered via - ``mask_secret()`` (e.g. connection passwords Airflow captured while - resolving the failing task's connections) -- it is not general-purpose - PII detection and will not catch arbitrary sensitive strings that were - never registered as secrets. Set to ``False`` only if you are certain - your exception messages contain no sensitive data and you need the - raw text for accurate classification. + - Whether to redact the exception's string representation before it is + added to the classification prompt. Set to ``False`` to disable + redaction entirely. Raises ``ValueError`` at construction time if + combined with an explicit ``redactor``. + * - ``max_exception_length`` + - 4096 + - Maximum number of characters of the (already redacted) exception + message included in the prompt. Longer messages are truncated with a + trailing ``"... (truncated)"`` marker. Must be a positive integer. + +Custom redactors +---------------- + +The default ``redactor`` only masks values already registered with Airflow's +secrets masker via ``mask_secret()``. It does not detect free-text PII -- +email addresses, customer names, account numbers -- that were never +registered as secrets. If your task's exception messages can contain that +kind of data, supply your own ``redactor`` callable. It **replaces** the +default masker rather than running in addition to it, so combine your own +logic with :func:`~airflow.providers.common.ai.policies.retry.redact_registered_secrets` +yourself if you still want known-secret masking too: + +.. code-block:: python + + import re + + from airflow.providers.common.ai.policies.retry import redact_registered_secrets + + EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+") + + + def redact_emails_and_secrets(message: str) -> str: + return redact_registered_secrets(EMAIL_RE.sub("", message)) + + + llm_policy = LLMRetryPolicy( + llm_conn_id="pydanticai_default", + redactor=redact_emails_and_secrets, + max_exception_length=2048, # keep long tracebacks from inflating token cost + ) + +To disable redaction entirely (for example, if you are certain your +exception messages contain no sensitive data and need the raw text for +accurate classification), pass ``redact_exception=False``: + +.. code-block:: python + + LLMRetryPolicy(llm_conn_id="pydanticai_default", redact_exception=False) Local LLM support ----------------- -By default, ``redact_exception`` already masks known secrets before the +By default, the built-in ``redactor`` already masks known secrets before the exception data reaches the LLM provider. For environments where exception data must not leave your own infrastructure at all -- even in masked form -- point to a local model via Ollama or vLLM instead, so the classification diff --git a/providers/common/ai/docs/toolsets.rst b/providers/common/ai/docs/toolsets.rst index a4b08bf869356..34a8b43f41727 100644 --- a/providers/common/ai/docs/toolsets.rst +++ b/providers/common/ai/docs/toolsets.rst @@ -128,7 +128,8 @@ Curated toolset wrapping * - ``get_schema`` - Returns column names and types for a table * - ``query`` - - Executes a SQL query and returns rows as JSON + - Executes a SQL query and returns bounded, columnar JSON (see + :ref:`bounded-query-results`) * - ``check_query`` - Validates SQL syntax without executing it @@ -203,6 +204,62 @@ Parameters Default ``False`` -- only SELECT-family and read-only metadata (``DESCRIBE``/``SHOW``) statements are permitted. - ``max_rows``: Maximum rows returned from the ``query`` tool. Default ``50``. + Rows beyond it are not read out of a DBAPI cursor; what the driver has already + transferred is its own call. See :ref:`bounded-query-results`. +- ``max_result_bytes``: Budget for the serialized ``query`` result. Default 64 KiB. + See :ref:`bounded-query-results`. + +.. _bounded-query-results: + +Bounded query results +^^^^^^^^^^^^^^^^^^^^^ + +A tool result stays in the model's message history for the rest of the run, so its +cost is re-paid on every subsequent model request. The ``query`` tool of both +``SQLToolset`` and ``DataFusionToolset`` bounds that in three ways. + +**The result is columnar.** Column names appear once, not once per row: + +.. code-block:: json + + {"columns": ["id", "name"], "rows": [[1, "Alice"], [2, "Bob"]], "row_count": 2} + +On a table with thousands of columns the repeated names, not the values, are the bulk +of a row-of-dicts payload. Positional rows also keep columns that share a name -- +``SELECT o.id, c.id`` -- which a dict per row silently collapsed to one. + +**Rows are fetched, not filtered.** ``max_rows`` bounds what leaves the cursor, so a +query matching a whole table costs the worker roughly what one matching ``max_rows`` +costs. How much is saved depends on the driver: with a server-side cursor the +remaining rows are never sent, while a client-buffering driver (psycopg2's default +cursor, MySQLdb) has already received them and only the per-row conversion is skipped. +Hooks whose cursor is not DBAPI 2.0 (``ExasolHook`` passes a pyexasol statement) fall +back to a full fetch, and ``DataFusionToolset`` materializes the full result in the +engine before the toolset sees it; in both the payload is bounded but the transfer is +not. + +**A byte budget bounds the payload.** ``max_rows`` caps rows, which says nothing about +size -- one row of a 3000-column table is larger than a thousand rows of a narrow one. +``max_result_bytes`` is what actually bounds context. Rows are returned as a contiguous +prefix: the result stops at the first row that does not fit the remaining budget rather +than skipping it and packing later ones, so a single wide row early in the result ends +it. The result says which limit it hit: + +.. code-block:: json + + {"columns": ["..."], "rows": ["..."], "row_count": 3, + "truncated": true, "truncated_by": "max_result_bytes"} + +``truncated_by`` is ``max_rows`` or ``max_result_bytes``. When not even one row fits, +or the column names alone exceed the budget, the result carries a ``hint`` telling the +agent to narrow its projection -- the only move that helps. ``total_rows`` is present +when the driver reports a row count for the query; several (SQLite, some warehouse +drivers) do not, and it is then omitted rather than guessed. + +The default budget is deliberately generous: the columnar shape alone shrinks a wide +result several-fold, so results that fit before still fit. Lower ``max_result_bytes`` +when an agent makes many queries in one run, since every result is re-paid on every +later request. ``DataFusionToolset`` --------------------- @@ -223,7 +280,8 @@ querying files on object stores (S3, local filesystem, Iceberg) via Apache DataF * - ``get_schema`` - Returns column names and types for a table (Arrow schema) * - ``query`` - - Executes a SQL query and returns rows as JSON + - Executes a SQL query and returns bounded, columnar JSON (see + :ref:`bounded-query-results`) Each :class:`~airflow.providers.common.sql.config.DataSourceConfig` entry registers a table backed by Parquet, CSV, Avro, or Iceberg data. Multiple @@ -267,6 +325,8 @@ Parameters permitted. DataFusion on object stores is mostly read-only, but it does support DDL for in-memory tables; this guard blocks those by default. - ``max_rows``: Maximum rows returned from the ``query`` tool. Default ``50``. +- ``max_result_bytes``: Budget for the serialized ``query`` result. Default 64 KiB. + See :ref:`bounded-query-results`. ``LoggingToolset`` ------------------ @@ -616,10 +676,14 @@ No single layer is sufficient — they work together. ``allowed_functions``. Fail-closed, but only as exact as the SQL parser. Not a security boundary -- always pair it with least-privilege database grants. See :ref:`allowed-tables-enforcement` below. - * - **SQLToolset: max_rows** - - Truncates query results to ``max_rows`` (default 50), preventing the - agent from pulling entire tables into context. - - Does not limit the number of queries the agent can make. + * - **SQLToolset: max_rows / max_result_bytes** + - Bounds a query result by rows (default 50) and by serialized size + (default 64 KiB), preventing the agent from pulling entire tables into + context. + - Does not limit the number of queries the agent can make, and each result + stays in message history for the rest of the run. Rows past ``max_rows`` + are not read out of the cursor, but a client-buffering driver has already + transferred them -- this bounds context, not database or network load. * - **MCPToolset: external server** - Connects the agent to tools exposed by an MCP server, authenticated through an Airflow connection. @@ -735,7 +799,8 @@ Recommended Configuration db_conn_id="analytics_readonly", # Connection with SELECT-only grants allowed_tables=["orders", "customers"], # Hide other tables from agent allow_writes=False, # Default — validates SQL - max_rows=50, # Default — truncate large results + max_rows=50, # Default — cap rows + max_result_bytes=65536, # Default — cap bytes; lower it for wide tables ) **Agents that need to modify data** (use with caution): @@ -763,8 +828,10 @@ Before deploying an agent task to production: agent can call any exposed tool with any arguments. 4. **Read-only default**: Keep ``allow_writes=False`` unless the task specifically requires writes. -5. **Row limits**: Set ``max_rows`` appropriate to the use case. Large - result sets consume LLM context and increase cost. +5. **Result limits**: Set ``max_rows`` and ``max_result_bytes`` appropriate to + the use case. ``max_rows`` alone does not bound size -- on wide tables it is + ``max_result_bytes`` that keeps a result from dominating the context window for + the rest of the run. 6. **Model budget**: Configure pydantic-ai's ``model_settings`` (e.g. ``max_tokens``) and ``retries`` to bound cost and prevent runaway loops. 7. **System prompt**: Include safety instructions in ``system_prompt`` (e.g. diff --git a/providers/common/ai/provider.yaml b/providers/common/ai/provider.yaml index 1feaa6ed5c685..b6133e2e2e458 100644 --- a/providers/common/ai/provider.yaml +++ b/providers/common/ai/provider.yaml @@ -147,6 +147,16 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook hook-name: "Pydantic AI" connection-type: pydanticai + external-services: + - OpenAI + - Anthropic + - Google + - AWS Bedrock + - Groq + - Mistral AI + - DeepSeek + - Ollama + - vLLM ui-field-behaviour: hidden-fields: - schema @@ -167,6 +177,8 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIAzureHook hook-name: "Pydantic AI (Azure OpenAI)" connection-type: pydanticai-azure + external-services: + - Azure OpenAI ui-field-behaviour: hidden-fields: - schema @@ -195,6 +207,8 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIBedrockHook hook-name: "Pydantic AI (AWS Bedrock)" connection-type: pydanticai-bedrock + external-services: + - AWS Bedrock ui-field-behaviour: hidden-fields: - schema @@ -278,6 +292,8 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIVertexHook hook-name: "Pydantic AI (Google Vertex AI)" connection-type: pydanticai-vertex + external-services: + - Google Vertex AI ui-field-behaviour: hidden-fields: - schema @@ -374,6 +390,14 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.langchain.LangChainHook hook-name: "LangChain" connection-type: langchain + external-services: + - OpenAI + - Anthropic + - Groq + - Mistral AI + - DeepSeek + - Ollama + - vLLM ui-field-behaviour: hidden-fields: - schema @@ -407,6 +431,10 @@ connection-types: - hook-class-name: airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook hook-name: "LlamaIndex" connection-type: llamaindex + external-services: + - OpenAI + - Ollama + - vLLM ui-field-behaviour: hidden-fields: - schema diff --git a/providers/common/ai/pyproject.toml b/providers/common/ai/pyproject.toml index e97151ecadb55..07a65f18dd469 100644 --- a/providers/common/ai/pyproject.toml +++ b/providers/common/ai/pyproject.toml @@ -137,7 +137,7 @@ dev = [ "apache-airflow-providers-standard", # Additional devel dependencies (do not remove this line and add extra development dependencies) "sqlglot>=30.0.0", - "pydantic-ai-slim[mcp]", + "pydantic-ai-slim[mcp]>=2.0.0", "pydantic-ai-skills>=1.2.0", "apache-airflow-providers-common-sql[datafusion]", "langchain>=1.0.0", diff --git a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_branch.py b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_branch.py index 664036ae1350d..f56f1c4ac0d01 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_branch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/decorators/llm_branch.py @@ -28,7 +28,10 @@ from typing import TYPE_CHECKING, Any, ClassVar from airflow.providers.common.ai.operators.llm_branch import LLMBranchOperator -from airflow.providers.common.ai.utils.validation import validate_prompt +from airflow.providers.common.ai.utils.validation import ( + reject_sequence_with_unsupported_feature, + validate_prompt, +) from airflow.providers.common.compat.sdk import ( DecoratedOperator, TaskDecorator, @@ -89,6 +92,12 @@ def execute(self, context: Context) -> Any: self.prompt = self.python_callable(*self.op_args, **kwargs) validate_prompt(self.prompt, decorator_name="@task.llm_branch") + reject_sequence_with_unsupported_feature( + self.prompt, + decorator_name="@task.llm_branch", + feature_name="require_approval", + feature_enabled=self.require_approval, + ) self.render_template_fields(context) return LLMBranchOperator.execute(self, context) diff --git a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_branch.py b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_branch.py index 9dceb68507169..c94e984a0113a 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_branch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_llm_branch.py @@ -18,6 +18,8 @@ from __future__ import annotations +from datetime import timedelta + from airflow.providers.common.ai.operators.llm_branch import LLMBranchOperator from airflow.providers.common.compat.sdk import dag, task @@ -150,3 +152,36 @@ def handle_packaging(): # [END howto_decorator_llm_branch_multi] example_llm_branch_decorator_multi() + + +# [START howto_operator_llm_branch_approval] +@dag(tags=["example"]) +def example_llm_branch_approval(): + route = LLMBranchOperator( + task_id="route_with_approval", + prompt="User says: 'I was charged twice for my subscription.'", + llm_conn_id="pydanticai_default", + system_prompt="Route support tickets to the right team.", + require_approval=True, + approval_timeout=timedelta(hours=24), + allow_modifications=True, + ) + + @task + def handle_billing(): + return "Handling billing issue" + + @task + def handle_auth(): + return "Handling auth issue" + + @task + def handle_general(): + return "Handling general issue" + + route >> [handle_billing(), handle_auth(), handle_general()] + + +# [END howto_operator_llm_branch_approval] + +example_llm_branch_approval() diff --git a/providers/common/ai/src/airflow/providers/common/ai/get_provider_info.py b/providers/common/ai/src/airflow/providers/common/ai/get_provider_info.py index 89d96737d94fa..96e87e5945d27 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/get_provider_info.py +++ b/providers/common/ai/src/airflow/providers/common/ai/get_provider_info.py @@ -120,6 +120,17 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", "hook-name": "Pydantic AI", "connection-type": "pydanticai", + "external-services": [ + "OpenAI", + "Anthropic", + "Google", + "AWS Bedrock", + "Groq", + "Mistral AI", + "DeepSeek", + "Ollama", + "vLLM", + ], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login"], "relabeling": {"password": "API Key"}, @@ -137,6 +148,7 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIAzureHook", "hook-name": "Pydantic AI (Azure OpenAI)", "connection-type": "pydanticai-azure", + "external-services": ["Azure OpenAI"], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login"], "relabeling": {"password": "API Key", "host": "Azure Endpoint"}, @@ -159,6 +171,7 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIBedrockHook", "hook-name": "Pydantic AI (AWS Bedrock)", "connection-type": "pydanticai-bedrock", + "external-services": ["AWS Bedrock"], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login", "host", "password"], "relabeling": {}, @@ -221,6 +234,7 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIVertexHook", "hook-name": "Pydantic AI (Google Vertex AI)", "connection-type": "pydanticai-vertex", + "external-services": ["Google Vertex AI"], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login", "host", "password"], "relabeling": {}, @@ -295,6 +309,15 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.langchain.LangChainHook", "hook-name": "LangChain", "connection-type": "langchain", + "external-services": [ + "OpenAI", + "Anthropic", + "Groq", + "Mistral AI", + "DeepSeek", + "Ollama", + "vLLM", + ], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login"], "relabeling": {"password": "API Key"}, @@ -319,6 +342,7 @@ def get_provider_info(): "hook-class-name": "airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook", "hook-name": "LlamaIndex", "connection-type": "llamaindex", + "external-services": ["OpenAI", "Ollama", "vLLM"], "ui-field-behaviour": { "hidden-fields": ["schema", "port", "login"], "relabeling": {"password": "API Key"}, diff --git a/providers/common/ai/src/airflow/providers/common/ai/mixins/approval.py b/providers/common/ai/src/airflow/providers/common/ai/mixins/approval.py index 9f045c0254888..cccff32bfd150 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/mixins/approval.py +++ b/providers/common/ai/src/airflow/providers/common/ai/mixins/approval.py @@ -17,6 +17,7 @@ from __future__ import annotations +import json import logging from datetime import timedelta from typing import TYPE_CHECKING, Any, Protocol @@ -45,6 +46,8 @@ class DeferForApprovalProtocol(Protocol): task_id: str defer: Any + def validate_approval_prompt(self) -> None: ... + class LLMApprovalMixin: """ @@ -70,6 +73,18 @@ class LLMApprovalMixin: APPROVE = "Approve" REJECT = "Reject" + def validate_approval_prompt(self: DeferForApprovalProtocol) -> None: + """Fail fast when the prompt cannot be rendered as text in the approval review body.""" + if not isinstance(self.prompt, str): + raise TypeError( + f"{type(self).__name__}: require_approval=True is not supported " + f"with a non-string prompt (got {type(self.prompt).__name__}). " + "The approval review body renders the prompt as text; passing a " + "Sequence[UserContent] would expose object reprs (and any embedded " + "bytes) in the human review UI. Return a str prompt, or disable " + "require_approval." + ) + def defer_for_approval( self: DeferForApprovalProtocol, context: Context, @@ -77,6 +92,7 @@ def defer_for_approval( *, subject: str | None = None, body: str | None = None, + modification_schema: dict[str, Any] | None = None, ) -> None: """ Write HITL detail, then pause the task for human review. @@ -91,20 +107,22 @@ def defer_for_approval( Defaults to ``"Review output for task ``"``. :param body: Markdown body shown below the headline. Defaults to the prompt and output wrapped in a code block. + :param modification_schema: JSON schema for the editable ``output`` param + when ``allow_modifications=True``. Defaults to ``{"type": "string"}``. + Pass e.g. ``{"type": "string", "enum": [...]}`` to render a dropdown + of valid values in the review form, or ``{"type": "array", "items": + {"type": "string", "enum": [...]}, "examples": [...]}`` to render a + multi-select (JSON Schema forbids ``enum`` at the array level, so the + options come from ``examples``); a list submitted by the reviewer is + returned from ``execute_complete`` re-serialized as a compact JSON string. """ from airflow.providers.standard.triggers.hitl import HITLTrigger from airflow.sdk.execution_time.hitl import upsert_hitl_detail from airflow.sdk.timezone import utcnow - if not isinstance(self.prompt, str): - raise TypeError( - "require_approval=True is not supported with a non-string prompt. " - "The approval review body renders the prompt as text; passing a " - "Sequence[UserContent] would expose object reprs (and any embedded " - "bytes) in the human review UI. Return a str prompt, or disable " - "require_approval." - ) + self.validate_approval_prompt() + raw_output = output if isinstance(output, BaseModel): output = output.model_dump_json() elif not isinstance(output, str): @@ -121,11 +139,19 @@ def defer_for_approval( hitl_params: dict[str, dict[str, Any]] = {} if self.allow_modifications: + # The multi-select rendered for an array schema needs the list, not its JSON string + param_value: Any = output + if ( + modification_schema is not None + and modification_schema.get("type") == "array" + and isinstance(raw_output, list) + ): + param_value = raw_output hitl_params = { "output": { - "value": output, + "value": param_value, "description": "Edit the output before approving (optional).", - "schema": {"type": "string"}, + "schema": modification_schema or {"type": "string"}, }, } @@ -203,13 +229,33 @@ def execute_complete(self, context: Context, generated_output: str, event: dict[ # when allow_modifications=False, bypassing the read-only approval flow. if getattr(self, "allow_modifications", False) and params_input: modified = params_input.get("output") + if "output" in params_input and modified is None: + raise HITLTriggerEventError( + { + "error": "Modified output must not be empty; edit it or reject instead.", + "error_type": "validation", + } + ) + if isinstance(modified, list): + for item in modified: + if not isinstance(item, str): + raise HITLTriggerEventError( + { + "error": f"Modified output list items must be strings, " + f"got {type(item).__name__}.", + "error_type": "validation", + } + ) + # Compact so an unchanged selection compares equal to generated_output + modified = json.dumps(modified, separators=(",", ":")) if modified is not None and not isinstance(modified, str): # On the awaiting_input path nothing upstream schema-validates params_input # (HITLTrigger did on the legacy path), so enforce the string contract here # rather than returning a non-string as the task's output. raise HITLTriggerEventError( { - "error": f"Modified output must be a string, got {type(modified).__name__}.", + "error": f"Modified output must be a string or a list of strings, " + f"got {type(modified).__name__}.", "error_type": "validation", } ) diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py index 95f33bffffaa7..36cd616c1b386 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm.py @@ -155,13 +155,8 @@ def llm_hook(self) -> PydanticAIHook: return PydanticAIHook.get_hook(self.llm_conn_id, hook_params=hook_params) def execute(self, context: Context) -> Any: - if self.require_approval and not isinstance(self.prompt, str): - raise TypeError( - f"{type(self).__name__}: require_approval=True is not supported " - f"with a non-string prompt (got {type(self.prompt).__name__}). " - f"The approval review body renders the prompt as text. Return a " - f"str prompt, or disable require_approval." - ) + if self.require_approval: + self.validate_approval_prompt() # type: ignore[misc] agent: Agent[object, Any] = self.llm_hook.create_agent( output_type=self.output_type, instructions=self.system_prompt, **self.agent_params diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py index b6a25a7811eb0..88c299432594b 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_branch.py @@ -18,6 +18,7 @@ from __future__ import annotations +import json from collections.abc import Iterable, Sequence from enum import Enum from typing import TYPE_CHECKING, Any @@ -47,6 +48,17 @@ class LLMBranchOperator(LLMOperator, BranchMixIn): single task ID. When ``True`` the LLM may return one or more task IDs. :param agent_params: Additional keyword arguments passed to the pydantic-ai ``Agent`` constructor (e.g. ``retries``, ``model_settings``, ``tools``). + + Human-in-the-Loop approval parameters are inherited from + :class:`~airflow.providers.common.ai.operators.llm.LLMOperator` + (``require_approval``, ``approval_timeout``, ``allow_modifications``). + The task pauses after the LLM chooses the branch(es) and only skips the + unselected downstream tasks once a reviewer approves. The review form + lists the valid downstream task IDs; with ``allow_modifications=True`` + the editable choice is rendered as a dropdown of those IDs (single-branch + mode) or a multi-select of them (``allow_multiple_branches=True``), and + the reviewed branch(es) are validated against the downstream task IDs + before branching. """ inherits_from_skipmixin = True @@ -60,12 +72,13 @@ def __init__( **kwargs: Any, ) -> None: kwargs.pop("output_type", None) - if kwargs.get("require_approval"): - raise ValueError("require_approval=True is not supported by LLMBranchOperator.") super().__init__(**kwargs) self.allow_multiple_branches = allow_multiple_branches def execute(self, context: Context) -> str | Iterable[str] | None: + if self.require_approval: + self.validate_approval_prompt() # type: ignore[misc] + if not self.downstream_task_ids: raise ValueError( f"{self.task_id!r} has no downstream tasks. " @@ -95,4 +108,61 @@ def execute(self, context: Context) -> str | Iterable[str] | None: else: branches = str(output) + if not branches: + raise ValueError( + f"LLM selected no branches for {self.task_id!r}, which would skip every downstream task." + ) + + if self.require_approval: + choices = sorted(self.downstream_task_ids) + chosen = branches if isinstance(branches, str) else json.dumps(branches) + body = ( + f"Valid branches: {', '.join(f'`{c}`' for c in choices)}\n\n" + f"```\nPrompt: {self.prompt}\n\nChosen branch(es): {chosen}\n```" + ) + modification_schema = ( + {"type": "array", "items": {"type": "string", "enum": choices}, "examples": choices} + if self.allow_multiple_branches + else {"type": "string", "enum": choices} + ) + self.defer_for_approval( # type: ignore[misc] + context, branches, body=body, modification_schema=modification_schema + ) + return self.do_branch(context, branches) + + def execute_complete(self, context: Context, generated_output: str, event: dict[str, Any]) -> Any: + """Resume after human review, validating the reviewed choice before branching.""" + output = super().execute_complete(context, generated_output, event) + branches = self._parse_reviewed_branches(output) + selected = {branches} if isinstance(branches, str) else set(branches) + invalid = selected - self.downstream_task_ids + if invalid: + raise ValueError( + f"Reviewed branch(es) {sorted(invalid)} are not downstream tasks of " + f"{self.task_id!r}. Valid choices: {sorted(self.downstream_task_ids)}." + ) + return self.do_branch(context, branches) + + def _parse_reviewed_branches(self, output: str) -> str | list[str]: + if not self.allow_multiple_branches: + return output + try: + branches = json.loads(output) + except json.JSONDecodeError as e: + raise ValueError( + f"Reviewed output {output!r} is not valid JSON. With " + f"allow_multiple_branches=True the reviewed output must be a " + f'JSON list of task IDs, e.g. ["task_a", "task_b"].' + ) from e + if not isinstance(branches, list) or not all(isinstance(b, str) for b in branches): + raise ValueError( + f"Reviewed output {output!r} must be a JSON list of task ID strings, " + f'e.g. ["task_a", "task_b"].' + ) + if not branches: + raise ValueError( + "Reviewed output selects no branches, which would skip every downstream " + "task. Select at least one task ID, or reject the review instead." + ) + return branches diff --git a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py index e8a3b327b4236..262d52ff8d0f4 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py +++ b/providers/common/ai/src/airflow/providers/common/ai/operators/llm_sql.py @@ -134,13 +134,8 @@ def db_hook(self) -> DbApiHook | None: return hook def execute(self, context: Context) -> str: - if self.require_approval and not isinstance(self.prompt, str): - raise TypeError( - f"{type(self).__name__}: require_approval=True is not supported " - f"with a non-string prompt (got {type(self.prompt).__name__}). " - f"The approval review body renders the prompt as text. Return a " - f"str prompt, or disable require_approval." - ) + if self.require_approval: + self.validate_approval_prompt() # type: ignore[misc] schema_info = self._get_schema_context() diff --git a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py index 768e16270e29d..864e5b6dad4f3 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py +++ b/providers/common/ai/src/airflow/providers/common/ai/policies/retry.py @@ -24,7 +24,7 @@ import logging from datetime import timedelta -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, cast from pydantic import BaseModel @@ -43,12 +43,14 @@ ) from None if TYPE_CHECKING: + from collections.abc import Callable + from airflow.sdk.definitions.context import Context from airflow.sdk.definitions.retry_policy import RetryRule log = logging.getLogger(__name__) -__all__ = ["ErrorClassification", "LLMRetryPolicy"] +__all__ = ["ErrorClassification", "LLMRetryPolicy", "redact_registered_secrets"] DEFAULT_INSTRUCTIONS = ( "You are an error classifier for a data pipeline system. " @@ -79,6 +81,12 @@ class ErrorClassification(BaseModel): """Brief explanation of the classification decision.""" +def redact_registered_secrets(message: str) -> str: + """Mask values registered via ``mask_secret()``; the default ``redactor`` for :class:`LLMRetryPolicy`.""" + # redact() is typed for arbitrary containers; a str in always yields a str out. + return cast("str", redact(message)) + + class LLMRetryPolicy(RetryPolicy): """ Retry policy that uses an LLM to classify errors and decide retry behaviour. @@ -102,12 +110,24 @@ class LLMRetryPolicy(RetryPolicy): falling back. Defaults to 30s. The LLM provider's own timeout (e.g. 600s for Anthropic) is much longer; this keeps the retry decision path fast even when the provider is degraded. - :param redact_exception: When ``True`` (the default), the exception's - string representation is passed through Airflow's secrets masker - (:func:`~airflow.sdk.log.redact`) before being added to the prompt. - Set to ``False`` only if you are certain your exception messages - contain no sensitive data and you need the raw text for accurate - classification. + :param redactor: Callable applied to the exception's string representation + before it is added to the classification prompt. Defaults to + :func:`~airflow.providers.common.ai.policies.retry.redact_registered_secrets`, + which only masks values already registered via ``mask_secret()``. + Pass a custom callable to replace the default masking entirely -- + for example to redact free-text PII the secrets masker cannot see. + To disable masking altogether, use ``redact_exception=False`` -- + not ``redactor=None``. + :param redact_exception: Whether to redact the exception's string + representation before it is added to the classification prompt. + Defaults to ``True``. Set to ``False`` to send the raw exception + text as-is. Passing ``redact_exception=False`` together with + an explicit ``redactor`` raises ``ValueError`` at construction time, + since the two settings would otherwise conflict silently. + :param max_exception_length: Maximum number of characters of the + (already redacted) exception message included in the prompt. Longer + messages are truncated with a trailing ``"... (truncated)"`` marker. + Must be a positive integer. Defaults to 4096. .. warning:: The exception's string representation is sent to the configured @@ -115,15 +135,18 @@ class LLMRetryPolicy(RetryPolicy): etc.) as part of the classification prompt, so it may leak whatever the failing task put in the exception message — connection strings, credential fragments, PII, or other secrets. By default - ``_classify()`` runs the message through Airflow's secrets masker - (:func:`~airflow.sdk.log.redact`, controlled by ``redact_exception``), - which masks values already registered via ``mask_secret()`` (for - example, connection passwords Airflow captured while resolving the - failing task's connections). This does **not** perform - general-purpose PII detection and will not catch arbitrary sensitive - strings that were never registered as secrets. You are still - responsible for confirming that your task's exception messages are - safe to send to a third-party LLM provider. + ``_classify()`` runs the message through + :func:`~airflow.providers.common.ai.policies.retry.redact_registered_secrets` + via ``redactor``, which masks values already registered via + ``mask_secret()`` (for example, connection passwords Airflow + captured while resolving the failing task's connections). This does + **not** perform general-purpose PII detection and will not catch + arbitrary sensitive strings that were never registered as secrets -- + for free-text PII (emails, customer names, etc.) supply your own + ``redactor``, or pass ``redact_exception=False`` to disable + redaction altogether. You are still responsible for confirming that + your task's exception messages are safe to send to a third-party + LLM provider. """ def __init__( @@ -134,14 +157,29 @@ def __init__( fallback_rules: list[RetryRule] | None = None, timeout: float = 30.0, *, + redactor: Callable[[str], str] | None = None, redact_exception: bool = True, + max_exception_length: int = 4096, ) -> None: + if max_exception_length <= 0: + raise ValueError(f"max_exception_length must be a positive integer, got {max_exception_length}") + if not redact_exception and redactor is not None: + raise ValueError( + "redactor must not be set when redact_exception=False -- passing an explicit " + "redactor while also disabling redaction is contradictory. Either drop " + "redact_exception=False to keep using redactor, or drop redactor to disable " + "redaction entirely." + ) self.llm_conn_id = llm_conn_id self.model_id = model_id self.instructions = instructions or DEFAULT_INSTRUCTIONS self.fallback_rules = fallback_rules self.timeout = timeout + self.redactor: Callable[[str], str] | None = ( + None if not redact_exception else redactor if redactor is not None else redact_registered_secrets + ) self.redact_exception = redact_exception + self.max_exception_length = max_exception_length def evaluate( self, @@ -174,11 +212,14 @@ def _classify( instructions=self.instructions, ) - exception_message = redact(str(exception)) if self.redact_exception else str(exception) + # Redact before truncating -- truncating first could cut a registered secret in half. + message = self.redactor(str(exception)) if self.redactor is not None else str(exception) + if len(message) > self.max_exception_length: + message = f"{message[: self.max_exception_length]}... (truncated)" prompt = ( f"Classify this error from a data pipeline task " f"(attempt {try_number} of {max_tries}):\n\n" - f"{type(exception).__name__}: {exception_message}" + f"{type(exception).__name__}: {message}" ) from pydantic_ai.settings import ModelSettings diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py index 78013918f8701..ef8881348115d 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/datafusion.py @@ -36,6 +36,11 @@ from pydantic_ai.tools import ToolDefinition from pydantic_ai.toolsets.abstract import AbstractToolset, ToolsetTool +from airflow.providers.common.ai.utils.query_results import ( + DEFAULT_MAX_RESULT_BYTES, + QUERY_TOOL_DESCRIPTION as _QUERY_DESCRIPTION, + build_query_result, +) from airflow.providers.common.ai.utils.tool_definition import build_args_validator if TYPE_CHECKING: @@ -97,6 +102,15 @@ class DataFusionToolset(AbstractToolset[Any]): are permitted. :param max_rows: Maximum number of rows returned from the ``query`` tool. Default ``50``. + :param max_result_bytes: Budget for the serialized ``query`` result, in bytes. + Default 64 KiB. ``max_rows`` bounds rows, which says nothing about size: one + row of a 3000-column table is larger than a thousand rows of a narrow one, and + a tool result stays in the model's message history for the rest of the run, so + its cost is re-paid on every subsequent request. Rows are returned as a + contiguous prefix, stopping at the first that does not fit the remaining budget + rather than skipping it and packing later ones, so one wide row early in the + result ends it. The result reports which limit it hit so the agent can narrow + its projection rather than page through the table. """ def __init__( @@ -105,12 +119,14 @@ def __init__( *, allow_writes: bool = False, max_rows: int = 50, + max_result_bytes: int = DEFAULT_MAX_RESULT_BYTES, ) -> None: if not datasource_configs: raise ValueError("datasource_configs must contain at least one DataSourceConfig") self._datasource_configs = datasource_configs self._allow_writes = allow_writes self._max_rows = max_rows + self._max_result_bytes = max_result_bytes self._engine: DataFusionEngine | None = None @property @@ -133,7 +149,7 @@ async def get_tools(self, ctx: RunContext[Any]) -> dict[str, ToolsetTool[Any]]: for name, description, schema in ( ("list_tables", "List available table names.", _LIST_TABLES_SCHEMA), ("get_schema", "Get column names and types for a table.", _GET_SCHEMA_SCHEMA), - ("query", "Execute a SQL query and return rows as JSON.", _QUERY_SCHEMA), + ("query", _QUERY_DESCRIPTION, _QUERY_SCHEMA), ): tool_def = ToolDefinition( name=name, @@ -199,16 +215,17 @@ def _query(self, sql: str) -> str: col_names = list(pydict.keys()) num_rows = len(next(iter(pydict.values()), [])) - result: list[dict[str, Any]] = [ - {col: pydict[col][i] for col in col_names} for i in range(min(num_rows, self._max_rows)) - ] - - truncated = num_rows > self._max_rows - output: dict[str, Any] = {"rows": result, "count": num_rows} - if truncated: - output["truncated"] = True - output["max_rows"] = self._max_rows - return json.dumps(output, default=str) + # DataFusion has already materialised the full result, so unlike SQLToolset + # there is nothing left to avoid fetching -- only the payload is bounded. + rows = [[pydict[col][i] for col in col_names] for i in range(min(num_rows, self._max_rows))] + return build_query_result( + col_names, + rows, + max_rows=self._max_rows, + max_result_bytes=self._max_result_bytes, + more_rows_available=num_rows > self._max_rows, + total_rows=num_rows, + ) except SQLSafetyError as ex: log.warning("query failed SQL safety validation: %s", ex) raise diff --git a/providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py b/providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py index f7b0f32001f09..584fd475c93aa 100644 --- a/providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py +++ b/providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py @@ -30,6 +30,7 @@ resolve_sqlglot_dialect, validate_sql as _validate_sql, ) + from airflow.providers.common.sql.hooks.handlers import get_row_count from airflow.providers.common.sql.hooks.sql import DbApiHook except ImportError as e: from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException @@ -40,6 +41,11 @@ from pydantic_ai.tools import ToolDefinition from pydantic_ai.toolsets.abstract import AbstractToolset, ToolsetTool +from airflow.providers.common.ai.utils.query_results import ( + DEFAULT_MAX_RESULT_BYTES, + QUERY_TOOL_DESCRIPTION as _QUERY_DESCRIPTION, + build_query_result, +) from airflow.providers.common.ai.utils.tool_definition import build_args_validator, return_schema_kwargs from airflow.providers.common.compat.sdk import BaseHook @@ -77,6 +83,75 @@ } +def _trusted_row_count(cursor: Any, *, fetched: int) -> int | None: + """ + Return the driver's row count for the query, or ``None`` when it cannot mean that. + + ``rowcount`` is only a query total on drivers that buffer the whole result before + handing back the first row. Others report rows fetched *so far* -- python-oracledb + documents exactly that for ``SELECT`` -- which after a capped fetch equals the cap, + not the total. Handing an agent ``total_rows: 51`` for a ten-million-row table is + worse than handing it nothing: it reads as authoritative and it is wrong. + + A count no larger than what was fetched is indistinguishable from that failure mode, + so it is discarded. Nothing is lost when the result was not truncated -- ``row_count`` + is already the total there. + """ + row_count = get_row_count(cursor) + if row_count is None or row_count <= fetched: + return None + return row_count + + +class _CappedFetch: + """ + ``DbApiHook.run`` handler that fetches at most ``limit`` rows instead of all of them. + + The counterpart in ``common.sql``, + :func:`~airflow.providers.common.sql.hooks.handlers.fetch_all_handler`, pulls the + whole result set into the worker; the toolset then discards all but ``max_rows`` of + it, having already paid for the transfer. Fetching through the cursor keeps the cost + proportional to what the agent is actually shown. + + How much this saves depends on the driver: with a server-side cursor the unfetched + rows are never sent, while a driver that buffers client-side (psycopg2's default + cursor, MySQLdb) has already received them and only the per-row conversion is + skipped. The result handed to the model is bounded either way. + + Not every hook hands its handler a DBAPI cursor -- ``ExasolHook`` passes a pyexasol + statement, which signals "this produced rows" through ``result_type`` rather than + ``description``. Bounding the fetch is not possible without knowing that per driver, + so those fall back to a full fetch, which is what the toolset did everywhere before. + The payload is still bounded; only the transfer is not. + + Instances are single-use -- ``total_rows`` refers to the last query run. + """ + + def __init__(self, limit: int) -> None: + self._limit = limit + #: Rows the driver reports for the query, or ``None`` when it reports none. + self.total_rows: int | None = None + + def __call__(self, cursor: Any) -> list[tuple] | None: + if not hasattr(cursor, "description"): + fetchall = getattr(cursor, "fetchall", None) + if not callable(fetchall): + raise RuntimeError( + "The database we interact with does not support DBAPI 2.0. Use a " + "connection whose hook exposes a DBAPI 2.0 cursor." + ) + rows = fetchall() + # Nothing was left behind, so the fetched count is the exact total. + self.total_rows = len(rows) if rows is not None else 0 + return rows + if cursor.description is None: + # A statement that returned no result set (DDL, or DML without RETURNING). + return None + rows = cursor.fetchmany(self._limit) + self.total_rows = _trusted_row_count(cursor, fetched=len(rows or [])) + return rows + + class SQLToolset(AbstractToolset[Any]): """ Curated toolset that gives an LLM agent safe access to a SQL database. @@ -143,7 +218,21 @@ class SQLToolset(AbstractToolset[Any]): :param allow_writes: Allow data-modifying SQL (INSERT, UPDATE, DELETE, etc.). Default ``False`` — only SELECT-family statements are permitted. :param max_rows: Maximum number of rows returned from the ``query`` tool. - Default ``50``. + Default ``50``. Rows beyond it are not pulled out of the cursor. How much that + saves is the driver's call, not this toolset's: a client-buffering driver + (psycopg2's default cursor, MySQLdb) has already received the whole result by + the time the first row is read, so only the per-row Python conversion is + skipped. Treat this as a bound on what the agent is shown, not as a guarantee + that ``SELECT * FROM huge_table`` is cheap. + :param max_result_bytes: Budget for the serialized ``query`` result, in bytes. + Default 64 KiB. ``max_rows`` bounds rows, which says nothing about size: one + row of a 3000-column table is larger than a thousand rows of a narrow one, and + a tool result stays in the model's message history for the rest of the run, so + its cost is re-paid on every subsequent request. Rows are returned as a + contiguous prefix, stopping at the first that does not fit the remaining budget + rather than skipping it and packing later ones, so one wide row early in the + result ends it. The result reports which limit it hit so the agent can narrow + its projection rather than page through the table. """ def __init__( @@ -155,6 +244,7 @@ def __init__( schema: str | None = None, allow_writes: bool = False, max_rows: int = 50, + max_result_bytes: int = DEFAULT_MAX_RESULT_BYTES, ) -> None: self._db_conn_id = db_conn_id self._allowed_tables: frozenset[str] | None = frozenset(allowed_tables) if allowed_tables else None @@ -166,6 +256,7 @@ def __init__( self._schema = schema self._allow_writes = allow_writes self._max_rows = max_rows + self._max_result_bytes = max_result_bytes self._hook: DbApiHook | None = None # Canonical ``(catalog, schema, table)`` view of allowed_tables for membership @@ -251,7 +342,7 @@ async def get_tools(self, ctx: RunContext[Any]) -> dict[str, ToolsetTool[Any]]: for name, description, schema in ( ("list_tables", "List available table names in the database.", _LIST_TABLES_SCHEMA), ("get_schema", "Get column names and types for a table.", _GET_SCHEMA_SCHEMA), - ("query", "Execute a SQL query and return rows as JSON.", _QUERY_SCHEMA), + ("query", _QUERY_DESCRIPTION, _QUERY_SCHEMA), ("check_query", "Validate SQL syntax without executing it.", _CHECK_QUERY_SCHEMA), ): # sequential=True because all tools use a shared DbApiHook with @@ -370,24 +461,24 @@ def _query(self, sql: str) -> str: if statements is not None: self._enforce_allowed_tables(statements) - rows = hook.get_records(sql) - # Fetch column names from cursor description. - col_names: list[str] | None = None + # One row beyond the cap, so "there is more" is knowable without fetching the + # rest. strip_sql_string mirrors what get_records did for the hooks that + # override it (Trino rejects a trailing semicolon) and is a no-op elsewhere. + fetch = _CappedFetch(self._max_rows + 1) + rows = hook.run(hook.strip_sql_string(sql), handler=fetch) or [] + + col_names: list[str] = [] if hook.last_description: col_names = [desc[0] for desc in hook.last_description] - result: list[dict[str, Any]] | list[list[Any]] - if rows and col_names: - result = [dict(zip(col_names, row)) for row in rows[: self._max_rows]] - else: - result = [list(row) for row in (rows or [])[: self._max_rows]] - - truncated = len(rows or []) > self._max_rows - output: dict[str, Any] = {"rows": result, "count": len(rows or [])} - if truncated: - output["truncated"] = True - output["max_rows"] = self._max_rows - return json.dumps(output, default=str) + return build_query_result( + col_names, + rows[: self._max_rows], + max_rows=self._max_rows, + max_result_bytes=self._max_result_bytes, + more_rows_available=len(rows) > self._max_rows, + total_rows=fetch.total_rows, + ) def _check_query(self, sql: str) -> str: # Resolve the dialect best-effort: if the connection can't be reached we diff --git a/providers/common/ai/src/airflow/providers/common/ai/utils/query_results.py b/providers/common/ai/src/airflow/providers/common/ai/utils/query_results.py new file mode 100644 index 0000000000000..956c9de86e336 --- /dev/null +++ b/providers/common/ai/src/airflow/providers/common/ai/utils/query_results.py @@ -0,0 +1,156 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" +Bounded, columnar payloads for the ``query`` tool of the SQL toolsets. + +A tool result stays in the model's message history for the rest of the run, so its +cost is re-paid on every subsequent request. Two things here keep that bounded: + +* **Columnar shape.** ``{"columns": [...], "rows": [[...], ...]}`` names each column + once instead of repeating it in a dict per row. On a table with thousands of + columns the repeated names, not the values, are the bulk of the payload. +* **A byte budget.** ``max_rows`` caps rows, which says nothing about size -- a + single row of a 3000-column table dwarfs a thousand rows of a narrow one. The + budget here is what actually bounds context, and when it bites the payload says + so, in terms the agent can act on (narrow the projection). +""" + +from __future__ import annotations + +import json +from collections.abc import Sequence +from typing import Any + +# A policy default, not a limit imposed by any storage, protocol, or model layer: +# roughly 16k tokens at 4 characters per token. Large enough that ordinary queries are +# unaffected, small enough that no single tool result can dominate the context window. +# Deployments that keep many results in history should lower it. +DEFAULT_MAX_RESULT_BYTES = 65_536 + +# Tool results are machine-read, so no whitespace. ensure_ascii=False matters as much as +# the separators: escaping one CJK character to \uXXXX costs six bytes instead of three, +# so an ASCII-escaped result is charged several times over against the budget and +# truncated that much earlier than an equivalent English one. +_DUMP_KWARGS: dict[str, Any] = {"default": str, "separators": (",", ":"), "ensure_ascii": False} + +#: Description for the ``query`` tool. States the columnar shape, since the model has +#: to align each row's values to ``columns`` positionally, and the truncation contract, +#: so a short result is not read as an empty table. +QUERY_TOOL_DESCRIPTION = ( + "Execute a SQL query. Returns JSON of the form " + '{"columns": [name, ...], "rows": [[value, ...], ...]}, where each row holds its ' + "values in column order. A `truncated` key means you are not seeing the whole result " + "-- either more rows matched than were returned, or the result was too large -- and " + "`truncated_by` names the limit that was hit; narrow the projection or aggregate in " + "SQL rather than paging through the result." +) + + +def _dumps(payload: Any) -> str: + return json.dumps(payload, **_DUMP_KWARGS) + + +def _size(payload: Any) -> int: + """Measure the serialized size in bytes -- not characters, which diverge outside ASCII.""" + return len(_dumps(payload).encode("utf-8")) + + +def build_query_result( + columns: Sequence[str], + rows: Sequence[Sequence[Any]], + *, + max_rows: int, + max_result_bytes: int, + more_rows_available: bool, + total_rows: int | None = None, +) -> str: + """ + Render query rows as a bounded, columnar JSON tool result. + + :param columns: Column names, in the order the values appear in each row. + :param rows: Rows already capped to ``max_rows``; only the byte budget is + applied here. + :param max_rows: The row cap that produced *rows*, reported back to the agent + so it knows which limit it hit. + :param max_result_bytes: Budget for the serialized column names plus rows. + The surrounding envelope (the ``truncated``/``hint`` keys) adds a small + fixed amount on top. + :param more_rows_available: Whether the query matched more rows than *rows* + holds. + :param total_rows: Total rows the driver reported for the query, when it + reports one at all. ``None`` is common and is not an error -- SQLite and + several warehouse drivers do not populate it for ``SELECT``. + """ + budget = max_result_bytes - _size(list(columns)) + if budget < 0: + # The column names alone blow the budget, so returning them would spend the + # whole context on a header and leave no room for data. Report the shape and + # tell the agent to narrow the projection -- the only move that helps here. + output: dict[str, Any] = { + "column_count": len(columns), + "rows": [], + "row_count": 0, + "truncated": True, + "truncated_by": "max_result_bytes", + "hint": ( + f"This query returns {len(columns)} column{'' if len(columns) == 1 else 's'}, " + f"whose names alone exceed max_result_bytes ({max_result_bytes}). Select the " + f"specific columns you need instead of all of them." + ), + } + if total_rows is not None: + output["total_rows"] = total_rows + return _dumps(output) + + # Rows are kept as a contiguous prefix: stopping at the first row that does not fit + # rather than skipping it and packing later ones, so "rows 1..n" means what it says + # and the agent is never handed a result with a hole in the middle. + kept: list[list[Any]] = [] + for row in rows: + as_list = list(row) + # +1 for the comma joining this row to the previous one; the serializer and + # separators match the final dump, so this accounting is exact. + cost = _size(as_list) + (1 if kept else 0) + if cost > budget: + break + budget -= cost + kept.append(as_list) + + byte_capped = len(kept) < len(rows) + output = {"columns": list(columns), "rows": kept, "row_count": len(kept)} + if total_rows is not None: + output["total_rows"] = total_rows + + if byte_capped or more_rows_available: + output["truncated"] = True + output["truncated_by"] = "max_result_bytes" if byte_capped else "max_rows" + if byte_capped: + # Say which row stopped it. "No row fits" would be false whenever a single wide + # row sits in front of narrow ones, and a partial result with no guidance at all + # is the common case, not the empty one. + output["hint"] = ( + f"The first row alone exceeds max_result_bytes ({max_result_bytes}). " + if not kept + else f"Stopped after {len(kept)} row{'' if len(kept) == 1 else 's'}: the next row " + f"did not fit in max_result_bytes ({max_result_bytes}). " + ) + "Select fewer columns, or aggregate, instead of returning whole rows." + elif more_rows_available: + output["hint"] = ( + f"Only the first {max_rows} rows are shown. Filter or aggregate in SQL rather " + f"than paging through the result." + ) + return _dumps(output) diff --git a/providers/common/ai/tests/unit/common/ai/decorators/test_llm_branch.py b/providers/common/ai/tests/unit/common/ai/decorators/test_llm_branch.py index 37238bff00b62..2243315dcf344 100644 --- a/providers/common/ai/tests/unit/common/ai/decorators/test_llm_branch.py +++ b/providers/common/ai/tests/unit/common/ai/decorators/test_llm_branch.py @@ -83,6 +83,25 @@ def test_execute_raises_on_invalid_prompt(self, return_value): with pytest.raises(TypeError, match="must be"): op.execute(context={}) + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_sequence_prompt_with_require_approval_raises_before_run_sync(self, mock_hook_cls): + """Sequence prompt + require_approval=True fails before the agent runs.""" + mock_agent = MagicMock(spec=["run_sync"]) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = _LLMBranchDecoratedOperator( + task_id="test", + python_callable=lambda: ["x", ImageUrl(url="https://example.com/x.png")], + llm_conn_id="my_llm", + require_approval=True, + ) + op.downstream_task_ids = {"positive"} + + with pytest.raises(TypeError, match="require_approval=True"): + op.execute(context={}) + + mock_agent.run_sync.assert_not_called() + @patch.object(LLMBranchOperator, "do_branch") @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) def test_execute_accepts_sequence_prompt(self, mock_hook_cls, mock_do_branch): diff --git a/providers/common/ai/tests/unit/common/ai/mixins/test_approval.py b/providers/common/ai/tests/unit/common/ai/mixins/test_approval.py index 46120b3c3c31f..ace3ce7d3d98e 100644 --- a/providers/common/ai/tests/unit/common/ai/mixins/test_approval.py +++ b/providers/common/ai/tests/unit/common/ai/mixins/test_approval.py @@ -146,6 +146,34 @@ def test_allow_modifications_creates_default_output_param( assert param["value"] == "draft text" assert param["schema"] == {"type": "string"} + @patch(HITL_TRIGGER_PATH, autospec=True) + @patch(UPSERT_HITL_PATH) + def test_modification_schema_overrides_output_param_schema( + self, mock_upsert, mock_trigger_cls, approval_op_with_modifications, context + ): + schema = {"type": "string", "enum": ["task_a", "task_b"]} + + approval_op_with_modifications.defer_for_approval(context, "task_a", modification_schema=schema) + + param = mock_upsert.call_args[1]["params"]["output"] + assert param["schema"] == schema + + @patch(HITL_TRIGGER_PATH, autospec=True) + @patch(UPSERT_HITL_PATH) + def test_array_schema_passes_list_param_value( + self, mock_upsert, mock_trigger_cls, approval_op_with_modifications, context + ): + choices = ["task_a", "task_b"] + schema = {"type": "array", "items": {"type": "string", "enum": choices}, "examples": choices} + + approval_op_with_modifications.defer_for_approval(context, ["task_a"], modification_schema=schema) + + param = mock_upsert.call_args[1]["params"]["output"] + assert param["value"] == ["task_a"] + assert param["schema"] == schema + defer_kwargs = approval_op_with_modifications.defer.call_args[1] + assert defer_kwargs["kwargs"]["generated_output"] == '["task_a"]' + @patch(HITL_TRIGGER_PATH, autospec=True) @patch(UPSERT_HITL_PATH) def test_no_modifications_params_empty(self, mock_upsert, mock_trigger_cls, approval_op, context): @@ -280,6 +308,52 @@ def test_approved_with_non_string_modified_output_raises(self, approval_op_with_ {}, generated_output="original output", event=event ) + def test_approved_with_list_modified_output_is_serialized(self, approval_op_with_modifications): + event = { + "chosen_options": ["Approve"], + "responded_by_user": "editor", + "params_input": {"output": ["task_b", "task_c"]}, + } + + result = approval_op_with_modifications.execute_complete( + {}, generated_output='["task_a"]', event=event + ) + + assert result == '["task_b","task_c"]' + + def test_approved_with_unmodified_list_output_returns_original(self, approval_op_with_modifications): + event = { + "chosen_options": ["Approve"], + "responded_by_user": "editor", + "params_input": {"output": ["task_a"]}, + } + + result = approval_op_with_modifications.execute_complete( + {}, generated_output='["task_a"]', event=event + ) + + assert result == '["task_a"]' + + def test_approved_with_non_string_list_items_raises(self, approval_op_with_modifications): + event = { + "chosen_options": ["Approve"], + "responded_by_user": "editor", + "params_input": {"output": ["task_a", 2]}, + } + + with pytest.raises(HITLTriggerEventError, match="items must be strings, got int"): + approval_op_with_modifications.execute_complete({}, generated_output='["task_a"]', event=event) + + def test_approved_with_cleared_output_raises(self, approval_op_with_modifications): + event = { + "chosen_options": ["Approve"], + "responded_by_user": "editor", + "params_input": {"output": None}, + } + + with pytest.raises(HITLTriggerEventError, match="must not be empty"): + approval_op_with_modifications.execute_complete({}, generated_output='["task_a"]', event=event) + def test_approved_with_unmodified_output(self, approval_op_with_modifications): event = { "chosen_options": ["Approve"], diff --git a/providers/common/ai/tests/unit/common/ai/operators/test_llm_branch.py b/providers/common/ai/tests/unit/common/ai/operators/test_llm_branch.py index fccf5e2f83737..7f2d11bf26336 100644 --- a/providers/common/ai/tests/unit/common/ai/operators/test_llm_branch.py +++ b/providers/common/ai/tests/unit/common/ai/operators/test_llm_branch.py @@ -18,11 +18,24 @@ from enum import Enum from unittest.mock import MagicMock, patch +from uuid import uuid4 import pytest +from airflow.providers.common.ai.mixins.approval import LLMApprovalMixin from airflow.providers.common.ai.operators.llm import LLMOperator from airflow.providers.common.ai.operators.llm_branch import LLMBranchOperator +from airflow.providers.common.compat.sdk import Param, ParamValidationError, TaskDeferred + +from tests_common.test_utils.version_compat import AIRFLOW_V_3_1_PLUS, AIRFLOW_V_3_3_PLUS + +if AIRFLOW_V_3_3_PLUS: + # On 3.3+ cores require_approval pauses the task in AWAITING_INPUT; older cores defer to + # HITLTrigger. Both signals carry method_name/kwargs/timeout, so the approval tests assert + # against whichever pause signal the running core uses. + from airflow.sdk.exceptions import TaskAwaitingInput as ApprovalPauseSignal +else: + ApprovalPauseSignal = TaskDeferred # type: ignore[assignment, misc] def _make_mock_run_result(output): @@ -54,15 +67,6 @@ def test_output_type_ignored(self): # the real output_type is built dynamically from downstream_task_ids assert op.output_type is str - def test_require_approval_rejected(self): - with pytest.raises(ValueError, match="require_approval=True is not supported"): - LLMBranchOperator( - task_id="test", - prompt="pick a branch", - llm_conn_id="my_llm", - require_approval=True, - ) - @patch.object(LLMBranchOperator, "do_branch") @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) def test_execute_single_branch(self, mock_hook_cls, mock_do_branch): @@ -117,6 +121,26 @@ def test_execute_multi_branch(self, mock_hook_cls, mock_do_branch): assert result == ["task_a", "task_c"] mock_do_branch.assert_called_once_with(ctx, ["task_a", "task_c"]) + @patch.object(LLMBranchOperator, "do_branch") + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_execute_rejects_empty_branch_selection(self, mock_hook_cls, mock_do_branch): + """LLM returning an empty list fails instead of skipping every downstream task.""" + mock_agent = MagicMock(spec=["run_sync"]) + mock_agent.run_sync.return_value = _make_mock_run_result([]) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="test", + prompt="Pick branches", + llm_conn_id="my_llm", + allow_multiple_branches=True, + ) + op.downstream_task_ids = {"task_a", "task_b"} + + with pytest.raises(ValueError, match="selected no branches"): + op.execute(MagicMock()) + mock_do_branch.assert_not_called() + @patch.object(LLMBranchOperator, "do_branch") @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) def test_system_prompt_forwarded(self, mock_hook_cls, mock_do_branch): @@ -173,3 +197,309 @@ def test_execute_raises_on_no_downstream_tasks(self): ) with pytest.raises(ValueError, match="no downstream tasks"): op.execute(MagicMock()) + + +def _make_context(ti_id=None): + ti_id = ti_id or uuid4() + ti = MagicMock() + ti.id = ti_id + return MagicMock(**{"__getitem__": lambda self, key: {"task_instance": ti}[key]}) + + +@pytest.mark.skipif( + not AIRFLOW_V_3_1_PLUS, reason="Human in the loop is only compatible with Airflow >= 3.1.0" +) +class TestLLMBranchOperatorApproval: + """Tests for LLMBranchOperator with require_approval=True (LLMApprovalMixin integration).""" + + def test_inherits_llm_approval_mixin(self): + assert issubclass(LLMBranchOperator, LLMApprovalMixin) + + @patch.object(LLMBranchOperator, "do_branch") + @patch("airflow.providers.standard.triggers.hitl.HITLTrigger", autospec=True) + @patch("airflow.sdk.execution_time.hitl.upsert_hitl_detail") + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_execute_with_approval_pauses_before_branching( + self, mock_hook_cls, mock_upsert, mock_trigger_cls, mock_do_branch + ): + """When require_approval=True, execute() pauses after the LLM choice, before do_branch.""" + downstream_enum = Enum("DownstreamTasks", {"task_a": "task_a", "task_b": "task_b"}) + + mock_agent = MagicMock(spec=["run_sync"]) + mock_agent.run_sync.return_value = _make_mock_run_result(downstream_enum.task_a) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="branch_approval", + prompt="Pick a branch", + llm_conn_id="my_llm", + require_approval=True, + ) + op.downstream_task_ids = {"task_a", "task_b"} + + with pytest.raises(ApprovalPauseSignal) as exc_info: + op.execute(_make_context()) + + assert exc_info.value.method_name == "execute_complete" + assert exc_info.value.kwargs["generated_output"] == "task_a" + mock_upsert.assert_called_once() + mock_do_branch.assert_not_called() + + @patch.object(LLMBranchOperator, "do_branch") + @patch("airflow.providers.standard.triggers.hitl.HITLTrigger", autospec=True) + @patch("airflow.sdk.execution_time.hitl.upsert_hitl_detail") + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_execute_with_approval_serializes_multiple_branches( + self, mock_hook_cls, mock_upsert, mock_trigger_cls, mock_do_branch + ): + """With allow_multiple_branches=True the choice is deferred as a JSON list.""" + downstream_enum = Enum("DownstreamTasks", {"task_a": "task_a", "task_c": "task_c"}) + + mock_agent = MagicMock(spec=["run_sync"]) + mock_agent.run_sync.return_value = _make_mock_run_result( + [downstream_enum.task_a, downstream_enum.task_c] + ) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="branch_approval_multi", + prompt="Pick branches", + llm_conn_id="my_llm", + allow_multiple_branches=True, + require_approval=True, + ) + op.downstream_task_ids = {"task_a", "task_b", "task_c"} + + with pytest.raises(ApprovalPauseSignal) as exc_info: + op.execute(_make_context()) + + assert exc_info.value.kwargs["generated_output"] == '["task_a","task_c"]' + mock_do_branch.assert_not_called() + + @patch.object(LLMBranchOperator, "do_branch") + @patch("airflow.providers.standard.triggers.hitl.HITLTrigger", autospec=True) + @patch("airflow.sdk.execution_time.hitl.upsert_hitl_detail") + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_review_form_lists_choices_and_renders_enum_dropdown( + self, mock_hook_cls, mock_upsert, mock_trigger_cls, mock_do_branch + ): + """The review body lists the valid branches and the editable param is an enum dropdown.""" + downstream_enum = Enum("DownstreamTasks", {"task_a": "task_a", "task_b": "task_b"}) + + mock_agent = MagicMock(spec=["run_sync"]) + mock_agent.run_sync.return_value = _make_mock_run_result(downstream_enum.task_a) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="branch_approval", + prompt="Pick a branch", + llm_conn_id="my_llm", + require_approval=True, + allow_modifications=True, + ) + op.downstream_task_ids = {"task_b", "task_a"} + + with pytest.raises(ApprovalPauseSignal): + op.execute(_make_context()) + + call_kwargs = mock_upsert.call_args.kwargs + assert call_kwargs["body"].startswith("Valid branches: `task_a`, `task_b`") + assert call_kwargs["params"]["output"]["schema"] == { + "type": "string", + "enum": ["task_a", "task_b"], + } + + @patch.object(LLMBranchOperator, "do_branch") + @patch("airflow.providers.standard.triggers.hitl.HITLTrigger", autospec=True) + @patch("airflow.sdk.execution_time.hitl.upsert_hitl_detail") + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_review_form_multi_branch_renders_multiselect( + self, mock_hook_cls, mock_upsert, mock_trigger_cls, mock_do_branch + ): + """With allow_multiple_branches the editable param is an array enum (multi-select).""" + downstream_enum = Enum("DownstreamTasks", {"task_a": "task_a", "task_b": "task_b"}) + + mock_agent = MagicMock(spec=["run_sync"]) + mock_agent.run_sync.return_value = _make_mock_run_result([downstream_enum.task_a]) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="branch_approval_multi", + prompt="Pick branches", + llm_conn_id="my_llm", + allow_multiple_branches=True, + require_approval=True, + allow_modifications=True, + ) + op.downstream_task_ids = {"task_a", "task_b"} + + with pytest.raises(ApprovalPauseSignal): + op.execute(_make_context()) + + call_kwargs = mock_upsert.call_args.kwargs + assert "Valid branches: `task_a`, `task_b`" in call_kwargs["body"] + assert call_kwargs["params"]["output"]["schema"] == { + "type": "array", + "items": {"type": "string", "enum": ["task_a", "task_b"]}, + "examples": ["task_a", "task_b"], + } + assert call_kwargs["params"]["output"]["value"] == ["task_a"] + + schema = call_kwargs["params"]["output"]["schema"] + assert Param(schema=schema).resolve(["task_a"]) == ["task_a"] + with pytest.raises(ParamValidationError): + Param(schema=schema).resolve(["task_x"]) + + @patch("airflow.providers.common.ai.operators.llm.PydanticAIHook", autospec=True) + def test_execute_rejects_sequence_prompt_with_require_approval(self, mock_hook_cls): + """Non-string prompt + require_approval=True fails before the agent runs.""" + mock_agent = MagicMock(spec=["run_sync"]) + mock_hook_cls.get_hook.return_value.create_agent.return_value = mock_agent + + op = LLMBranchOperator( + task_id="test", + prompt=["describe", b"bytes"], # type: ignore[arg-type] + llm_conn_id="my_llm", + require_approval=True, + ) + op.downstream_task_ids = {"task_a"} + + with pytest.raises(TypeError, match="require_approval=True"): + op.execute(_make_context()) + + mock_agent.run_sync.assert_not_called() + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_approved_single_branch(self, mock_do_branch): + """execute_complete branches into the approved task.""" + mock_do_branch.return_value = "task_a" + op = LLMBranchOperator(task_id="t", prompt="p", llm_conn_id="c") + op.downstream_task_ids = {"task_a", "task_b"} + event = {"chosen_options": ["Approve"], "responded_by_user": "admin"} + ctx = _make_context() + + result = op.execute_complete(ctx, generated_output="task_a", event=event) + + assert result == "task_a" + mock_do_branch.assert_called_once_with(ctx, "task_a") + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_approved_multiple_branches(self, mock_do_branch): + """execute_complete parses the JSON list back before branching.""" + mock_do_branch.return_value = ["task_a", "task_c"] + op = LLMBranchOperator(task_id="t", prompt="p", llm_conn_id="c", allow_multiple_branches=True) + op.downstream_task_ids = {"task_a", "task_b", "task_c"} + event = {"chosen_options": ["Approve"], "responded_by_user": "admin"} + ctx = _make_context() + + result = op.execute_complete(ctx, generated_output='["task_a","task_c"]', event=event) + + assert result == ["task_a", "task_c"] + mock_do_branch.assert_called_once_with(ctx, ["task_a", "task_c"]) + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_with_modified_branch(self, mock_do_branch): + """A reviewer-modified branch is used when it is a valid downstream task.""" + mock_do_branch.return_value = "task_b" + op = LLMBranchOperator(task_id="t", prompt="p", llm_conn_id="c", allow_modifications=True) + op.downstream_task_ids = {"task_a", "task_b"} + event = { + "chosen_options": ["Approve"], + "responded_by_user": "admin", + "params_input": {"output": "task_b"}, + } + ctx = _make_context() + + result = op.execute_complete(ctx, generated_output="task_a", event=event) + + assert result == "task_b" + mock_do_branch.assert_called_once_with(ctx, "task_b") + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_with_multiselect_modified_branches(self, mock_do_branch): + """A list submitted by the multi-select review form branches into those tasks.""" + mock_do_branch.return_value = ["task_b", "task_c"] + op = LLMBranchOperator( + task_id="t", + prompt="p", + llm_conn_id="c", + allow_multiple_branches=True, + allow_modifications=True, + ) + op.downstream_task_ids = {"task_a", "task_b", "task_c"} + event = { + "chosen_options": ["Approve"], + "responded_by_user": "admin", + "params_input": {"output": ["task_b", "task_c"]}, + } + ctx = _make_context() + + result = op.execute_complete(ctx, generated_output='["task_a"]', event=event) + + assert result == ["task_b", "task_c"] + mock_do_branch.assert_called_once_with(ctx, ["task_b", "task_c"]) + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_rejects_invalid_modified_branch(self, mock_do_branch): + """A reviewer-modified branch outside downstream_task_ids fails validation.""" + op = LLMBranchOperator(task_id="t", prompt="p", llm_conn_id="c", allow_modifications=True) + op.downstream_task_ids = {"task_a", "task_b"} + event = { + "chosen_options": ["Approve"], + "responded_by_user": "admin", + "params_input": {"output": "task_x"}, + } + + with pytest.raises(ValueError, match="not downstream tasks"): + op.execute_complete(_make_context(), generated_output="task_a", event=event) + + mock_do_branch.assert_not_called() + + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_rejects_empty_branch_list(self, mock_do_branch): + """A reviewed empty list would skip every downstream task and must be rejected.""" + op = LLMBranchOperator( + task_id="t", + prompt="p", + llm_conn_id="c", + allow_multiple_branches=True, + allow_modifications=True, + ) + op.downstream_task_ids = {"task_a", "task_b"} + event = { + "chosen_options": ["Approve"], + "responded_by_user": "admin", + "params_input": {"output": "[]"}, + } + + with pytest.raises(ValueError, match="selects no branches"): + op.execute_complete(_make_context(), generated_output='["task_a"]', event=event) + + mock_do_branch.assert_not_called() + + @pytest.mark.parametrize( + "modified", + ["not json", '{"task_a": 1}', '["task_a", 2]'], + ids=["malformed-json", "not-a-list", "non-string-item"], + ) + @patch.object(LLMBranchOperator, "do_branch") + def test_execute_complete_rejects_invalid_multi_branch_shapes(self, mock_do_branch, modified): + """With allow_multiple_branches=True the reviewed output must be a JSON list of strings.""" + op = LLMBranchOperator( + task_id="t", + prompt="p", + llm_conn_id="c", + allow_multiple_branches=True, + allow_modifications=True, + ) + op.downstream_task_ids = {"task_a", "task_b"} + event = { + "chosen_options": ["Approve"], + "responded_by_user": "admin", + "params_input": {"output": modified}, + } + + with pytest.raises(ValueError, match="JSON list"): + op.execute_complete(_make_context(), generated_output='["task_a"]', event=event) + + mock_do_branch.assert_not_called() diff --git a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py index 6175938e07a10..9b38eb243e1e1 100644 --- a/providers/common/ai/tests/unit/common/ai/policies/test_retry.py +++ b/providers/common/ai/tests/unit/common/ai/policies/test_retry.py @@ -28,6 +28,7 @@ from airflow.providers.common.ai.policies.retry import ( ErrorClassification, LLMRetryPolicy, + redact_registered_secrets, ) from airflow.sdk._shared.secrets_masker import reset_secrets_masker from airflow.sdk.definitions.retry_policy import RetryAction, RetryRule @@ -48,6 +49,18 @@ def _make_mock_agent(category, should_retry, delay=0, reasoning="test"): return mock_agent +@pytest.mark.enable_redact +def test_redact_registered_secrets_masks_only_registered_values(): + """Docs tell Dag authors to import and wrap this, so both the name and its narrow scope are contracts.""" + reset_secrets_masker() + mask_secret("super-secret-conn-password") + + assert ( + redact_registered_secrets("contact user@example.com with super-secret-conn-password") + == "contact user@example.com with ***" + ) + + class TestLLMClassifyDecisions: """Test that _classify maps LLM classification to correct RetryDecisions.""" @@ -154,6 +167,115 @@ def test_prompt_keeps_raw_message_when_redaction_disabled(self, mock_hook_cls): prompt = mock_agent.run_sync.call_args[0][0] assert secret_value in prompt + @pytest.mark.enable_redact + @patch("airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", autospec=True) + def test_explicit_redactor_none_still_applies_default_masking(self, mock_hook_cls): + """redactor=None means "use the default masker" -- the same as omitting it.""" + reset_secrets_masker() + secret_value = "super-secret-conn-password" + mask_secret(secret_value) + + mock_agent = _make_mock_agent("auth", should_retry=False) + mock_hook_cls.return_value.create_agent.return_value = mock_agent + + policy = LLMRetryPolicy(llm_conn_id="test", redactor=None) + policy.evaluate( + ConnectionError(f"could not authenticate with password {secret_value}"), + try_number=1, + max_tries=3, + ) + + prompt = mock_agent.run_sync.call_args[0][0] + assert secret_value not in prompt + assert "***" in prompt + + def test_redact_exception_false_with_explicit_redactor_raises(self): + with pytest.raises(ValueError, match="redactor must not be set when redact_exception=False"): + LLMRetryPolicy(llm_conn_id="test", redact_exception=False, redactor=lambda message: message) + + @pytest.mark.enable_redact + @patch("airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", autospec=True) + def test_custom_redactor_replaces_masker_instead_of_stacking(self, mock_hook_cls): + """A custom redactor replaces the secrets masker entirely -- it is not applied on top.""" + reset_secrets_masker() + secret_value = "super-secret-conn-password" + mask_secret(secret_value) + + mock_agent = _make_mock_agent("auth", should_retry=False) + mock_hook_cls.return_value.create_agent.return_value = mock_agent + + policy = LLMRetryPolicy(llm_conn_id="test", redactor=lambda s: s.replace("authenticate", "REDACTED")) + policy.evaluate( + ConnectionError(f"could not authenticate with password {secret_value}"), + try_number=1, + max_tries=3, + ) + + prompt = mock_agent.run_sync.call_args[0][0] + # The registered secret is untouched by the masker... + assert secret_value in prompt + # ...but the custom redactor's own transformation did apply. + assert "REDACTED" in prompt + + @pytest.mark.parametrize( + ("max_exception_length", "message_length", "expect_truncated"), + [ + pytest.param(4096, 4096, False, id="default-limit-exact-fit"), + pytest.param(4096, 5000, True, id="default-limit-exceeded"), + pytest.param(10, 20, True, id="custom-limit-exceeded"), + pytest.param(10, 5, False, id="custom-limit-under"), + ], + ) + @patch("airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", autospec=True) + def test_message_truncated_when_over_max_exception_length( + self, mock_hook_cls, max_exception_length, message_length, expect_truncated + ): + mock_agent = _make_mock_agent("data", should_retry=False) + mock_hook_cls.return_value.create_agent.return_value = mock_agent + + policy = LLMRetryPolicy( + llm_conn_id="test", redact_exception=False, max_exception_length=max_exception_length + ) + policy.evaluate(ValueError("x" * message_length), try_number=1, max_tries=3) + + prompt = mock_agent.run_sync.call_args[0][0] + assert ("... (truncated)" in prompt) is expect_truncated + if expect_truncated: + assert f"{'x' * max_exception_length}... (truncated)" in prompt + else: + assert "x" * message_length in prompt + + @pytest.mark.enable_redact + @patch("airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", autospec=True) + def test_truncation_happens_after_redaction(self, mock_hook_cls): + """Redact-then-truncate must not equal truncate-then-redact for this input. + + The secret sits right at the truncation boundary: truncating first would slice + it in half so the masker could no longer recognize and mask it. + """ + reset_secrets_masker() + secret_value = "super-secret-conn-password" + mask_secret(secret_value) + max_exception_length = 20 + # Padding places the secret so it straddles the truncation boundary. + padding = "a" * (max_exception_length - 5) + message = f"{padding}{secret_value}" + + mock_agent = _make_mock_agent("auth", should_retry=False) + mock_hook_cls.return_value.create_agent.return_value = mock_agent + + policy = LLMRetryPolicy(llm_conn_id="test", max_exception_length=max_exception_length) + policy.evaluate(ConnectionError(message), try_number=1, max_tries=3) + + prompt = mock_agent.run_sync.call_args[0][0] + assert secret_value not in prompt + assert "***" in prompt + + @pytest.mark.parametrize("max_exception_length", [0, -1, -100]) + def test_non_positive_max_exception_length_raises(self, max_exception_length): + with pytest.raises(ValueError, match="max_exception_length must be a positive integer"): + LLMRetryPolicy(llm_conn_id="test", max_exception_length=max_exception_length) + @patch("airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook", autospec=True) def test_custom_instructions_forwarded_to_agent(self, mock_hook_cls): mock_hook_cls.return_value.create_agent.return_value = _make_mock_agent("x", False) diff --git a/providers/common/ai/tests/unit/common/ai/toolsets/test_datafusion.py b/providers/common/ai/tests/unit/common/ai/toolsets/test_datafusion.py index 5bcf240afaaa7..05481efa115dd 100644 --- a/providers/common/ai/tests/unit/common/ai/toolsets/test_datafusion.py +++ b/providers/common/ai/tests/unit/common/ai/toolsets/test_datafusion.py @@ -180,8 +180,9 @@ def test_returns_rows_as_json(self): ) ) data = json.loads(result) - assert data["rows"] == [{"id": 1, "amount": 10.5}, {"id": 2, "amount": 20.0}] - assert data["count"] == 2 + assert data["columns"] == ["id", "amount"] + assert data["rows"] == [[1, 10.5], [2, 20.0]] + assert data["row_count"] == 2 def test_truncates_at_max_rows(self): cfg = _make_mock_datasource_config() @@ -197,9 +198,10 @@ def test_truncates_at_max_rows(self): ) ) data = json.loads(result) - assert len(data["rows"]) == 1 + assert data["rows"] == [[1, "a"]] assert data["truncated"] is True - assert data["count"] == 3 + assert data["truncated_by"] == "max_rows" + assert data["total_rows"] == 3 def test_handles_empty_result(self): cfg = _make_mock_datasource_config() @@ -216,7 +218,7 @@ def test_handles_empty_result(self): ) data = json.loads(result) assert data["rows"] == [] - assert data["count"] == 0 + assert data["row_count"] == 0 def test_blocks_create_table_by_default(self): cfg = _make_mock_datasource_config() diff --git a/providers/common/ai/tests/unit/common/ai/toolsets/test_sql.py b/providers/common/ai/tests/unit/common/ai/toolsets/test_sql.py index 9ff50ce142c70..0a00ef9c04a62 100644 --- a/providers/common/ai/tests/unit/common/ai/toolsets/test_sql.py +++ b/providers/common/ai/tests/unit/common/ai/toolsets/test_sql.py @@ -21,33 +21,76 @@ from unittest.mock import MagicMock, PropertyMock, patch import pytest +from pydantic_ai._run_context import RunContext from pydantic_ai.exceptions import ModelRetry +from pydantic_ai.toolsets.abstract import ToolsetTool from pydantic_core import ValidationError +from sqlalchemy.engine.reflection import Inspector from airflow.providers.common.ai.toolsets.sql import SQLToolset from airflow.providers.common.ai.utils.tool_definition import _SUPPORTS_RETURN_SCHEMA from airflow.providers.common.sql.hooks.sql import DbApiHook +class _FakeCursor: + """ + Minimal DBAPI cursor over canned records. + + The toolset fetches through ``DbApiHook.run``'s handler protocol rather than + ``get_records``, so the mock hook drives the real handler over this instead of + returning a canned list -- that keeps ``fetchmany``'s bounded-fetch behaviour + under test rather than mocked away. + """ + + def __init__(self, records: list[tuple], description: list[tuple] | None, rowcount: int) -> None: + self.description = description + self.rowcount = rowcount + self._remaining = list(records) + + def fetchmany(self, size: int) -> list[tuple]: + chunk, self._remaining = self._remaining[:size], self._remaining[size:] + return chunk + + def _make_mock_db_hook( table_names: list[str] | None = None, table_schema: list[dict[str, str]] | None = None, records: list[tuple] | None = None, last_description: list[tuple] | None = None, + rowcount: int = -1, ): """Create a mock DbApiHook with sensible defaults.""" mock = MagicMock(spec=DbApiHook) - mock.inspector = MagicMock() + mock.inspector = MagicMock(spec=Inspector) mock.inspector.get_table_names.return_value = table_names or ["users", "orders"] mock.get_table_schema.return_value = table_schema or [ {"name": "id", "type": "INTEGER"}, {"name": "name", "type": "VARCHAR"}, ] - mock.get_records.return_value = records or [(1, "Alice"), (2, "Bob")] - type(mock).last_description = PropertyMock(return_value=last_description or [("id",), ("name",)]) + rows = [(1, "Alice"), (2, "Bob")] if records is None else records + description = last_description or [("id",), ("name",)] + # -1 is what a driver reports when it has no row count for a SELECT (SQLite); + # get_row_count normalises that to None, so total_rows is absent by default. + mock.strip_sql_string.side_effect = DbApiHook.strip_sql_string + mock.run.side_effect = lambda sql, handler, **kwargs: handler(_FakeCursor(rows, description, rowcount)) + type(mock).last_description = PropertyMock(return_value=description) return mock +def _assert_executed(hook, sql: str) -> None: + """Assert the toolset ran exactly *sql* (the handler is an implementation detail).""" + assert hook.run.call_count == 1 + assert hook.run.call_args.args[0] == sql + + +def _query(ts: SQLToolset, sql: str) -> dict: + """Run the ``query`` tool and decode its JSON result.""" + result = asyncio.run( + ts.call_tool("query", {"sql": sql}, ctx=MagicMock(spec=RunContext), tool=MagicMock(spec=ToolsetTool)) + ) + return json.loads(result) + + class TestSQLToolsetInit: def test_id_includes_conn_id(self): ts = SQLToolset("my_pg") @@ -163,19 +206,30 @@ def test_introspection_error_raises_model_retry(self): class TestSQLToolsetQuery: - def test_returns_rows_as_json(self): + def test_returns_rows_columnar(self): + """Column names are named once, not repeated per row -- on a wide table the + repeated names, not the values, dominate the payload.""" ts = SQLToolset("pg_default") ts._hook = _make_mock_db_hook( records=[(1, "Alice"), (2, "Bob")], last_description=[("id",), ("name",)], ) - result = asyncio.run( - ts.call_tool("query", {"sql": "SELECT id, name FROM users"}, ctx=MagicMock(), tool=MagicMock()) - ) - data = json.loads(result) - assert data["rows"] == [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}] - assert data["count"] == 2 + data = _query(ts, "SELECT id, name FROM users") + assert data["columns"] == ["id", "name"] + assert data["rows"] == [[1, "Alice"], [2, "Bob"]] + assert data["row_count"] == 2 + assert "truncated" not in data + + def test_duplicate_column_names_are_preserved(self): + """A dict per row silently dropped one of two same-named columns; positional + rows keep both.""" + ts = SQLToolset("pg_default") + ts._hook = _make_mock_db_hook(records=[(1, 2)], last_description=[("id",), ("id",)]) + + data = _query(ts, "SELECT o.id, c.id FROM orders o JOIN customers c ON o.id = c.id") + assert data["columns"] == ["id", "id"] + assert data["rows"] == [[1, 2]] def test_truncates_at_max_rows(self): ts = SQLToolset("pg_default", max_rows=1) @@ -184,13 +238,158 @@ def test_truncates_at_max_rows(self): last_description=[("id",), ("name",)], ) - result = asyncio.run( - ts.call_tool("query", {"sql": "SELECT id, name FROM users"}, ctx=MagicMock(), tool=MagicMock()) + data = _query(ts, "SELECT id, name FROM users") + assert data["rows"] == [[1, "Alice"]] + assert data["row_count"] == 1 + assert data["truncated"] is True + assert data["truncated_by"] == "max_rows" + + def test_fetches_only_one_row_beyond_max_rows(self): + """The rows past the cap are never pulled out of the cursor. Fetching them all + and then slicing pays the full transfer cost for data the agent never sees.""" + cursor = _FakeCursor([(i, "x") for i in range(1000)], [("id",), ("name",)], -1) + ts = SQLToolset("pg_default", max_rows=2) + ts._hook = _make_mock_db_hook() + ts._hook.run.side_effect = lambda sql, handler, **kwargs: handler(cursor) + + _query(ts, "SELECT id, name FROM users") + # 1000 rows matched; 3 left the cursor (max_rows plus the one that proves + # there are more), so 997 were never transferred. + assert len(cursor._remaining) == 997 + + def test_reports_total_rows_when_the_driver_provides_one(self): + ts = SQLToolset("pg_default", max_rows=1) + ts._hook = _make_mock_db_hook( + records=[(1, "Alice"), (2, "Bob")], + last_description=[("id",), ("name",)], + rowcount=4_200_000, ) - data = json.loads(result) - assert len(data["rows"]) == 1 + + assert _query(ts, "SELECT id, name FROM users")["total_rows"] == 4_200_000 + + def test_omits_total_rows_when_the_driver_counts_rows_fetched_so_far(self): + """python-oracledb documents rowcount for SELECT as rows fetched *so far*, so + after a capped fetch it equals the cap, not the query total. Reporting it would + tell an agent it saw 50 of 51 rows when the table holds millions.""" + ts = SQLToolset("pg_default", max_rows=50) + ts._hook = _make_mock_db_hook( + records=[(i, f"name_{i}") for i in range(500)], + last_description=[("id",), ("name",)], + # Fetching max_rows + 1 leaves an incremental driver reporting exactly 51. + rowcount=51, + ) + + data = _query(ts, "SELECT id, name FROM big") + assert data["row_count"] == 50 assert data["truncated"] is True - assert data["count"] == 3 + assert "total_rows" not in data + + def test_omits_total_rows_when_the_driver_has_none(self): + """SQLite and several warehouse drivers report -1 for a SELECT. Absent beats + inventing a count the agent would treat as authoritative.""" + ts = SQLToolset("pg_default") + ts._hook = _make_mock_db_hook(records=[(1, "Alice")], rowcount=-1) + + assert "total_rows" not in _query(ts, "SELECT id, name FROM users") + + def test_wide_rows_are_capped_by_byte_budget_before_max_rows(self): + """max_rows says nothing about size. A handful of very wide rows must still be + bounded, or the result dominates the context for the rest of the run.""" + ts = SQLToolset("pg_default", max_rows=50, max_result_bytes=16_384) + ts._hook = _make_mock_db_hook( + records=[tuple(f"value_{i}" for i in range(200))] * 50, + last_description=[(f"col_{i}",) for i in range(200)], + ) + + data = _query(ts, "SELECT * FROM wide") + # 50 rows are under max_rows, so only the byte budget can have stopped this. + assert 0 < data["row_count"] < 50 + assert len(json.dumps(data, separators=(",", ":"))) <= 16_384 + 512 # budget + envelope + assert data["truncated"] is True + assert data["truncated_by"] == "max_result_bytes" + + def test_row_too_wide_to_fit_tells_the_agent_to_narrow_the_projection(self): + ts = SQLToolset("pg_default", max_result_bytes=200) + ts._hook = _make_mock_db_hook(records=[("x" * 500,)], last_description=[("blob",)]) + + data = _query(ts, "SELECT blob FROM docs") + assert data["rows"] == [] + assert data["truncated_by"] == "max_result_bytes" + assert "max_result_bytes" in data["hint"] + + def test_column_names_alone_over_budget_report_the_shape_instead(self): + """When even the header does not fit, returning it would spend the whole budget + on column names and leave no room for data.""" + ts = SQLToolset("pg_default", max_result_bytes=256) + ts._hook = _make_mock_db_hook( + records=[tuple(range(3000))], + last_description=[(f"column_name_{i}",) for i in range(3000)], + ) + + data = _query(ts, "SELECT * FROM very_wide") + assert "columns" not in data + assert data["column_count"] == 3000 + assert data["rows"] == [] + assert "3000 columns" in data["hint"] + + def test_no_rows_matched(self): + ts = SQLToolset("pg_default") + ts._hook = _make_mock_db_hook(records=[], last_description=[("id",), ("name",)]) + + data = _query(ts, "SELECT id, name FROM users WHERE 1=0") + assert data == {"columns": ["id", "name"], "rows": [], "row_count": 0} + + def test_statement_returning_no_result_set(self): + """A DBAPI cursor reports description=None for DDL and for DML without + RETURNING; that is not an error, just no rows.""" + ts = SQLToolset("pg_default", allow_writes=True) + ts._hook = _make_mock_db_hook() + ts._hook.run.side_effect = lambda sql, handler, **kwargs: handler(_FakeCursor([], None, -1)) + + assert _query(ts, "INSERT INTO users VALUES (3, 'Eve')")["row_count"] == 0 + + def test_non_dbapi_cursor_falls_back_to_a_full_fetch(self): + """ExasolHook hands its handler a pyexasol statement, which has no + ``description``. Bounding the fetch needs driver-specific knowledge, so those + fall back to fetching everything -- as the toolset did before -- rather than + failing the query. The payload is still bounded.""" + + class _NonDbapiStatement: + def __init__(self, records): + self._records = records + + def fetchall(self): + return self._records + + ts = SQLToolset("pg_default", max_rows=2) + ts._hook = _make_mock_db_hook() + ts._hook.run.side_effect = lambda sql, handler, **kwargs: handler( + _NonDbapiStatement([(i, f"name_{i}") for i in range(10)]) + ) + + data = _query(ts, "SELECT id, name FROM users") + assert data["rows"] == [[0, "name_0"], [1, "name_1"]] + assert data["truncated_by"] == "max_rows" + # A full fetch left nothing behind, so the count is exact and worth reporting. + assert data["total_rows"] == 10 + + def test_cursor_that_can_neither_describe_nor_fetch_is_an_error(self): + """Returning an empty result would read to the agent as 'the table is empty'.""" + ts = SQLToolset("pg_default") + ts._hook = _make_mock_db_hook() + ts._hook.run.side_effect = lambda sql, handler, **kwargs: handler(object()) + + with pytest.raises(ModelRetry, match="DBAPI 2.0"): + _query(ts, "SELECT id FROM users") + + def test_trailing_semicolon_is_stripped(self): + """get_records did this for the hooks that override it -- Trino rejects a + trailing semicolon -- so fetching through run() has to keep doing it.""" + ts = SQLToolset("pg_default") + ts._hook = _make_mock_db_hook() + + _query(ts, "SELECT id FROM users;") + _assert_executed(ts._hook, "SELECT id FROM users") def test_unsafe_sql_raises_model_retry(self): """An unsafe statement is surfaced to the agent as a retry so it can switch to a SELECT.""" @@ -232,7 +431,7 @@ def test_query_error_is_returned_to_agent_as_model_retry(self, error): max_retries bounds the loop, so an unrecoverable error still fails the task.""" ts = SQLToolset("pg_default") ts._hook = _make_mock_db_hook() - ts._hook.get_records.side_effect = error + ts._hook.run.side_effect = error with pytest.raises(ModelRetry) as exc_info: asyncio.run( @@ -448,7 +647,7 @@ def test_describe_allowed_through_query(self): ) data = json.loads(result) assert "rows" in data - ts._hook.get_records.assert_called_once_with("DESCRIBE TABLE users") + _assert_executed(ts._hook, "DESCRIBE TABLE users") def test_show_allowed_with_snowflake_dialect(self): """SHOW parses to a metadata statement once the hook's dialect is passed through.""" @@ -459,7 +658,7 @@ def test_show_allowed_with_snowflake_dialect(self): result = asyncio.run(ts.call_tool("query", {"sql": "SHOW TABLES"}, ctx=MagicMock(), tool=MagicMock())) data = json.loads(result) assert "rows" in data - ts._hook.get_records.assert_called_once_with("SHOW TABLES") + _assert_executed(ts._hook, "SHOW TABLES") @pytest.mark.parametrize( "sql", @@ -477,7 +676,7 @@ def test_query_blocks_disallowed_statements(self, sql): with pytest.raises(ModelRetry) as exc_info: asyncio.run(ts.call_tool("query", {"sql": sql}, ctx=MagicMock(), tool=MagicMock())) assert "not allowed" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_check_query_accepts_describe(self): ts = SQLToolset("pg_default") @@ -518,7 +717,7 @@ def test_query_allows_table_on_the_list(self): result = _run_query(ts, "SELECT id FROM orders") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once_with("SELECT id FROM orders") + _assert_executed(ts._hook, "SELECT id FROM orders") def test_query_blocks_table_off_the_list(self): """The headline escape: querying a table that is not on the allow-list is refused.""" @@ -530,7 +729,7 @@ def test_query_blocks_table_off_the_list(self): assert "not in the allowed tables list" in exc_info.value.message assert "secret_salaries" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() @pytest.mark.parametrize( "sql", @@ -551,7 +750,7 @@ def test_query_blocks_disallowed_table_reached_indirectly(self, sql): _run_query(ts, sql) assert "secret_salaries" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_blocks_catalog_enumeration(self): """information_schema/pg_catalog are ordinary tables, so the allow-list blocks them too.""" @@ -562,7 +761,7 @@ def test_query_blocks_catalog_enumeration(self): _run_query(ts, "SELECT table_name FROM information_schema.tables") assert "information_schema.tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_allows_cte_reference_not_mistaken_for_table(self): """A CTE whose name is not on the list is fine as long as its body stays allowed.""" @@ -583,7 +782,7 @@ def test_query_blocks_table_valued_function(self): _run_query(ts, "SELECT * FROM dblink('host=evil', 'SELECT 1') AS t(x int)") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() @pytest.mark.parametrize( "sql", @@ -608,7 +807,7 @@ def test_query_blocks_data_reaching_functions(self, sql): _run_query(ts, sql) assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_blocks_copy_from_program_under_allow_writes(self): """COPY ... FROM PROGRAM is OS command execution. Even with allow_writes=True (only @@ -622,7 +821,7 @@ def test_query_blocks_copy_from_program_under_allow_writes(self): _run_query(ts, "COPY orders FROM PROGRAM 'id > /tmp/x'") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_check_query_reports_data_reaching_function_invalid(self): """check_query surfaces the same rejection so the agent learns before executing.""" @@ -644,7 +843,7 @@ def test_query_allows_typed_builtins_over_allowed_table(self): result = _run_query(ts, "SELECT count(*), lower(name) FROM orders") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once_with("SELECT count(*), lower(name) FROM orders") + _assert_executed(ts._hook, "SELECT count(*), lower(name) FROM orders") def test_query_blocks_unrecognized_function_by_default(self): """Fail-closed: a legit-but-unrecognized builtin (json_build_object) is refused @@ -657,7 +856,7 @@ def test_query_blocks_unrecognized_function_by_default(self): _run_query(ts, "SELECT json_build_object('id', id) FROM orders") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_allows_function_named_in_allowed_functions(self): """allowed_functions is the opt-in escape hatch for a safe unrecognized function.""" @@ -668,7 +867,7 @@ def test_query_allows_function_named_in_allowed_functions(self): result = _run_query(ts, "SELECT json_build_object('id', id) FROM orders") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once_with("SELECT json_build_object('id', id) FROM orders") + _assert_executed(ts._hook, "SELECT json_build_object('id', id) FROM orders") def test_allowed_functions_does_not_permit_a_dangerous_sibling(self): """Permitting one function does not open the door to another unlisted one.""" @@ -680,7 +879,7 @@ def test_allowed_functions_does_not_permit_a_dangerous_sibling(self): _run_query(ts, "SELECT json_build_object('x', pg_read_file('/etc/passwd')) FROM orders") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_blocks_show_when_allowlist_active(self): ts = SQLToolset("sf_default", allowed_tables=["orders"]) @@ -691,7 +890,7 @@ def test_query_blocks_show_when_allowlist_active(self): _run_query(ts, "SHOW TABLES") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_blocks_describe_of_disallowed_table(self): ts = SQLToolset("sf_default", allowed_tables=["orders"]) @@ -702,7 +901,7 @@ def test_query_blocks_describe_of_disallowed_table(self): _run_query(ts, "DESCRIBE TABLE secret_salaries") assert "secret_salaries" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_query_allows_describe_of_allowed_table(self): ts = SQLToolset("sf_default", allowed_tables=["orders"]) @@ -712,7 +911,7 @@ def test_query_allows_describe_of_allowed_table(self): result = _run_query(ts, "DESCRIBE TABLE orders") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once_with("DESCRIBE TABLE orders") + _assert_executed(ts._hook, "DESCRIBE TABLE orders") def test_query_allows_schema_qualified_table_on_list(self): ts = SQLToolset("sf", allowed_tables=["MODEL_CRM.SF_ASTRO_ORGS"]) @@ -740,7 +939,7 @@ def test_no_allowlist_leaves_queries_unrestricted(self): result = _run_query(ts, "SELECT * FROM anything_at_all") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once_with("SELECT * FROM anything_at_all") + _assert_executed(ts._hook, "SELECT * FROM anything_at_all") def test_check_query_reports_disallowed_table_as_invalid(self): ts = SQLToolset("pg_default", allowed_tables=["orders"]) @@ -764,14 +963,14 @@ def test_writes_still_bounded_by_allowed_tables(self): # An allowed target is written. _run_query(ts, "INSERT INTO orders (id) VALUES (1)") - ts._hook.get_records.assert_called_once_with("INSERT INTO orders (id) VALUES (1)") + _assert_executed(ts._hook, "INSERT INTO orders (id) VALUES (1)") # A disallowed target is refused before execution. - ts._hook.get_records.reset_mock() + ts._hook.run.reset_mock() with pytest.raises(ModelRetry) as exc_info: _run_query(ts, "INSERT INTO secret_salaries (id) VALUES (1)") assert "secret_salaries" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_writes_reject_dynamic_sql_the_parser_cannot_inspect(self): """allow_writes skips the read-only validator, so the allow-list must still @@ -784,7 +983,7 @@ def test_writes_reject_dynamic_sql_the_parser_cannot_inspect(self): _run_query(ts, "EXEC sp_who") assert "cannot be checked against allowed_tables" in exc_info.value.message - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() class TestSQLToolsetAllowedTablesBypassRegressions: @@ -854,7 +1053,7 @@ def test_known_bypasses_are_rejected(self, sql, dialect, allow_writes): with pytest.raises(ModelRetry): _run_query(ts, sql) - ts._hook.get_records.assert_not_called() + ts._hook.run.assert_not_called() def test_legit_cte_over_allowed_table_still_runs(self): """The scope-aware fix must not false-reject a genuine CTE over an allowed table.""" @@ -864,7 +1063,7 @@ def test_legit_cte_over_allowed_table_still_runs(self): result = _run_query(ts, "WITH ranked AS (SELECT * FROM orders) SELECT * FROM ranked") assert "rows" in json.loads(result) - ts._hook.get_records.assert_called_once() + assert ts._hook.run.call_count == 1 def test_dml_with_cte_source_over_allowed_table_runs(self): """A CTE used as a DML source must not be mistaken for a disallowed base table.""" @@ -874,4 +1073,4 @@ def test_dml_with_cte_source_over_allowed_table_runs(self): sql = "WITH src AS (SELECT * FROM orders) INSERT INTO orders SELECT * FROM src" _run_query(ts, sql) - ts._hook.get_records.assert_called_once_with(sql) + _assert_executed(ts._hook, sql) diff --git a/providers/common/ai/tests/unit/common/ai/utils/test_query_results.py b/providers/common/ai/tests/unit/common/ai/utils/test_query_results.py new file mode 100644 index 0000000000000..0fb33b3b8023c --- /dev/null +++ b/providers/common/ai/tests/unit/common/ai/utils/test_query_results.py @@ -0,0 +1,162 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import datetime +import json +from decimal import Decimal + +import pytest + +from airflow.providers.common.ai.utils.query_results import build_query_result + + +def _build(columns, rows, *, max_rows=50, max_result_bytes=65_536, more=False, total=None) -> dict: + return json.loads( + build_query_result( + columns, + rows, + max_rows=max_rows, + max_result_bytes=max_result_bytes, + more_rows_available=more, + total_rows=total, + ) + ) + + +class TestShape: + def test_columns_are_named_once(self): + data = _build(["id", "name"], [(1, "a"), (2, "b")]) + assert data == {"columns": ["id", "name"], "rows": [[1, "a"], [2, "b"]], "row_count": 2} + + def test_columnar_is_smaller_than_a_dict_per_row(self): + """The saving that motivates the shape: on a wide table the repeated column + names, not the values, are the bulk of the payload.""" + columns = [f"column_name_{i}" for i in range(500)] + rows = [tuple(range(500))] * 10 + + columnar = build_query_result( + columns, rows, max_rows=50, max_result_bytes=10**9, more_rows_available=False + ) + per_row_dicts = json.dumps( + {"rows": [dict(zip(columns, row)) for row in rows], "count": 10}, + separators=(",", ":"), + ) + # The header is serialized once rather than once per row, so the gap widens + # with row count; at 10 rows it is already a factor of three. + assert len(columnar) * 3 < len(per_row_dicts) + + def test_non_json_types_are_stringified(self): + data = _build( + ["at", "amount"], + [(datetime.datetime(2026, 8, 8, 12, 0), Decimal("1.50"))], + ) + assert data["rows"] == [["2026-08-08 12:00:00", "1.50"]] + + +class TestTotalRows: + def test_included_when_known(self): + assert _build(["id"], [(1,)], total=97)["total_rows"] == 97 + + def test_absent_when_the_driver_reports_none(self): + assert "total_rows" not in _build(["id"], [(1,)], total=None) + + +class TestByteBudget: + def test_budget_is_accounted_exactly(self): + """The per-row measurement uses the same serializer and separators as the final + dump, so the emitted columns-plus-rows must land inside the budget.""" + columns = [f"col_{i}" for i in range(20)] + rows = [tuple(f"value_{i}" for i in range(20))] * 100 + budget = 4096 + + data = _build(columns, rows, max_rows=100, max_result_bytes=budget) + measured = len(json.dumps(data["columns"], separators=(",", ":"))) + len( + json.dumps(data["rows"], separators=(",", ":")) + ) + row_size = len(json.dumps(list(rows[0]), separators=(",", ":"))) + + # Never over budget (the +2 is the rows array's own brackets, which the per-row + # accounting does not charge for) and never more than one unplaced row under it. + assert measured <= budget + 2 + assert measured > budget - row_size + assert data["truncated_by"] == "max_result_bytes" + + def test_max_rows_wins_when_it_bites_first(self): + data = _build(["id"], [(1,)], more=True) + assert data["truncated"] is True + assert data["truncated_by"] == "max_rows" + + def test_a_single_oversized_row_yields_a_narrowing_hint(self): + data = _build(["blob"], [("x" * 400,)], max_result_bytes=100) + assert data["rows"] == [] + assert data["truncated_by"] == "max_result_bytes" + assert "first row alone exceeds max_result_bytes (100)" in data["hint"] + + def test_an_oversized_row_stops_the_result_rather_than_being_skipped(self): + """Packing later rows around a wide one would hand the agent a prefix with a + hole in it. Stopping keeps 'rows 1..n' meaning what it says.""" + rows = [("small",), ("x" * 5000,), ("small",), ("small",)] + + data = _build(["blob"], rows, max_result_bytes=1000) + assert data["rows"] == [["small"]] + assert data["truncated_by"] == "max_result_bytes" + + def test_partial_byte_truncation_still_guides_the_agent(self): + """The partial case is the common one; it used to carry no hint at all, so the + agent saw a short result with no reason to change its query.""" + rows = [("small",), ("x" * 5000,), ("small",)] + + data = _build(["blob"], rows, max_result_bytes=1000) + assert "Stopped after 1 row:" in data["hint"] + assert "Select fewer columns" in data["hint"] + + def test_budget_counts_bytes_not_escape_sequences(self): + """With ensure_ascii the budget charges six characters per CJK character, so a + Japanese result would be truncated several times earlier than an English one + carrying the same information.""" + rows = [("日本語のテキスト",)] * 40 + + data = _build(["text"], rows, max_result_bytes=2048) + assert data["rows"][0] == ["日本語のテキスト"] + # Three bytes per character, not the six an escaped \uXXXX would cost. + assert data["row_count"] > 40 / 2 + + def test_reported_size_is_bytes_for_non_ascii(self): + rows = [("é" * 200,)] + data = _build(["text"], rows, max_result_bytes=300) + # 400 bytes of payload cannot fit a 300-byte budget, though it is 200 characters. + assert data["rows"] == [] + + def test_columns_over_budget_report_the_shape_without_the_names(self): + columns = [f"column_name_{i}" for i in range(3000)] + data = _build(columns, [tuple(range(3000))], max_result_bytes=512, total=9) + + assert "columns" not in data + assert data["column_count"] == 3000 + assert data["row_count"] == 0 + assert data["total_rows"] == 9 + assert "3000 columns" in data["hint"] + + @pytest.mark.parametrize("max_result_bytes", [0, -1]) + def test_non_positive_budget_returns_no_rows_rather_than_raising(self, max_result_bytes): + data = _build(["id"], [(1,)], max_result_bytes=max_result_bytes) + assert data["row_count"] == 0 + + def test_empty_result_is_not_reported_as_truncated(self): + data = _build(["id", "name"], []) + assert data == {"columns": ["id", "name"], "rows": [], "row_count": 0} diff --git a/providers/common/compat/docs/changelog.rst b/providers/common/compat/docs/changelog.rst index 919c2b980ecfd..4e88aebbf0e88 100644 --- a/providers/common/compat/docs/changelog.rst +++ b/providers/common/compat/docs/changelog.rst @@ -436,7 +436,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 1.4.0 ..... diff --git a/providers/common/sql/README.rst b/providers/common/sql/README.rst index f04f36b720d66..aa37c6d293ea1 100644 --- a/providers/common/sql/README.rst +++ b/providers/common/sql/README.rst @@ -84,10 +84,10 @@ Dependent package Optional dependencies ---------------------- -================== ================================================================================================================================================================================ +================== ======================================================================================================================================================================= Extra Dependencies -================== ================================================================================================================================================================================ -``pandas`` ``pandas[sql-other]>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +================== ======================================================================================================================================================================= +``pandas`` ``pandas[sql-other]>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``openlineage`` ``apache-airflow-providers-openlineage`` ``polars`` ``polars>=1.26.0`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` @@ -95,7 +95,7 @@ Extra Dependencies ``datafusion`` ``datafusion>=50.0.0,<52.0.0`` ``pyiceberg-core`` ``pyiceberg-core>=0.8.0`` ``apache.iceberg`` ``apache-airflow-providers-apache-iceberg`` -================== ================================================================================================================================================================================ +================== ======================================================================================================================================================================= The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/common/sql/docs/changelog.rst b/providers/common/sql/docs/changelog.rst index 787ec8ec15501..93d01d7bdb5bf 100644 --- a/providers/common/sql/docs/changelog.rst +++ b/providers/common/sql/docs/changelog.rst @@ -43,11 +43,12 @@ Bug Fixes Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): * ``Use common.compat.sdk for timezone imports in providers (#70492)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 2.0.3 ..... @@ -577,7 +578,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 1.22.0 ...... diff --git a/providers/common/sql/docs/index.rst b/providers/common/sql/docs/index.rst index ff00456d28e48..ae215cc5a7022 100644 --- a/providers/common/sql/docs/index.rst +++ b/providers/common/sql/docs/index.rst @@ -141,10 +141,10 @@ Install them when installing from PyPI. For example: pip install apache-airflow-providers-common-sql[pandas] -================== ================================================================================================================================================================================ +================== ======================================================================================================================================================================= Extra Dependencies -================== ================================================================================================================================================================================ -``pandas`` ``pandas[sql-other]>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +================== ======================================================================================================================================================================= +``pandas`` ``pandas[sql-other]>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``openlineage`` ``apache-airflow-providers-openlineage`` ``polars`` ``polars>=1.26.0`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` @@ -152,7 +152,7 @@ Extra Dependencies ``datafusion`` ``datafusion>=50.0.0,<52.0.0`` ``pyiceberg-core`` ``pyiceberg-core>=0.8.0`` ``apache.iceberg`` ``apache-airflow-providers-apache-iceberg`` -================== ================================================================================================================================================================================ +================== ======================================================================================================================================================================= Downloading official packages ----------------------------- diff --git a/providers/common/sql/pyproject.toml b/providers/common/sql/pyproject.toml index b018c3cca45d4..184f0e10073c8 100644 --- a/providers/common/sql/pyproject.toml +++ b/providers/common/sql/pyproject.toml @@ -72,20 +72,11 @@ dependencies = [ # Any change in the dependencies is preserved when the file is regenerated [project.optional-dependencies] "pandas" = [ - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas[sql-other]>=2.1.2,<3; python_version <"3.13"', + 'pandas[sql-other]>=2.1.2; python_version <"3.13"', # Technically - we should add "sql-other" here as well, but this will only be possible when we move # to sqlalchemy 2+ TODO(potiuk): do so. - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] "openlineage" = [ "apache-airflow-providers-openlineage" diff --git a/providers/databricks/README.rst b/providers/databricks/README.rst index 712ed795418b1..e51f00031c479 100644 --- a/providers/databricks/README.rst +++ b/providers/databricks/README.rst @@ -50,9 +50,9 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.13.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` @@ -60,13 +60,13 @@ PIP package Version required ``databricks-sql-connector`` ``>=4.4.0`` ``aiohttp`` ``>=3.14.0,<4`` ``mergedeep`` ``>=1.3.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyarrow`` ``>=16.1.0; python_version < "3.13"`` ``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` -========================================== ==================================================================== +========================================== ================================================================== Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/databricks/docs/changelog.rst b/providers/databricks/docs/changelog.rst index 41f025336ef34..02005033d3acf 100644 --- a/providers/databricks/docs/changelog.rst +++ b/providers/databricks/docs/changelog.rst @@ -32,7 +32,6 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Review and update constraint dependencies (#70652)`` * ``Validate DatabricksSQLStatementsSensor statement fields after rendering (#70340)`` * ``Validate DatabricksCopyIntoOperator template fields after rendering (#70339)`` @@ -47,6 +46,8 @@ Doc-only appropriate section above if needed. Do not delete the lines(!): * ``Restore Dag-parse-time validation for Databricks Repos operator arguments (#70551)`` * ``Validate Databricks Repos operators' template fields after rendering (#70341)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 7.18.0 ...... @@ -705,7 +706,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 7.1.0 ..... diff --git a/providers/databricks/docs/index.rst b/providers/databricks/docs/index.rst index 75f80e9b4542a..67e283f16c8ef 100644 --- a/providers/databricks/docs/index.rst +++ b/providers/databricks/docs/index.rst @@ -98,9 +98,9 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.13.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` @@ -108,13 +108,13 @@ PIP package Version required ``databricks-sql-connector`` ``>=4.4.0`` ``aiohttp`` ``>=3.14.0,<4`` ``mergedeep`` ``>=1.3.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyarrow`` ``>=16.1.0; python_version < "3.13"`` ``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` -========================================== ==================================================================== +========================================== ================================================================== Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/databricks/docs/operators/notebook.rst b/providers/databricks/docs/operators/notebook.rst index 83dd899b41279..3c8ec27b8d38b 100644 --- a/providers/databricks/docs/operators/notebook.rst +++ b/providers/databricks/docs/operators/notebook.rst @@ -42,3 +42,41 @@ Running a notebook in Databricks on an existing cluster :language: python :start-after: [START howto_operator_databricks_notebook_existing_cluster] :end-before: [END howto_operator_databricks_notebook_existing_cluster] + +Configuring Databricks-native task retries +------------------------------------------- + +Use ``max_retries``, ``min_retry_interval_millis`` and ``retry_on_timeout`` to configure +`Databricks-native task retries `_. +Databricks reruns failed task attempts within the same job run, so Airflow sees only the final result. +Set ``max_retries`` to ``-1`` to retry indefinitely, or ``0`` to disable retries. + +These settings are independent of the Airflow task-level ``retries`` parameter, which retries the +whole Airflow task: + +.. code-block:: python + + DatabricksNotebookOperator( + task_id="notebook", + notebook_path="/path/to/notebook", + source="WORKSPACE", + existing_cluster_id="existing_cluster_id", + max_retries=3, + min_retry_interval_millis=2000, + retry_on_timeout=True, + ) + +Airflow ``retries`` behaves differently depending on where the operator runs. For a standalone +operator, each retry submits a new Databricks run. Inside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +the Airflow task monitors a sub-run that was already submitted by the workflow launch task, so a +retry only re-polls the terminal sub-run. Use ``max_retries`` to retry Databricks work inside a +workflow task group. + +Inside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +a task that exhausts a finite ``max_retries`` is reported as failed as soon as its final failed +attempt is observed, so downstream failure handling is not delayed by long-running sibling tasks. +Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the +parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt +under the same ``task_key`` until then. Sibling tasks in the run continue independently. diff --git a/providers/databricks/docs/operators/run_now.rst b/providers/databricks/docs/operators/run_now.rst index 00ee6ed0fecfe..156049444e6a4 100644 --- a/providers/databricks/docs/operators/run_now.rst +++ b/providers/databricks/docs/operators/run_now.rst @@ -60,7 +60,14 @@ for this run. If ``job_parameters`` is not set in ``json`` and the operator's ``params`` dict is non-empty, ``params`` is forwarded as ``job_parameters`` as-is, so Airflow Dag params can be passed dynamically to a run without hardcoding them in ``json``. If ``json`` already -contains ``job_parameters``, it is left untouched. +contains ``job_parameters``, it is left untouched. You can set ``forward_dag_params=False`` to +disable this parameter forwarding behavior. + +.. note:: + The Databricks API does not permit ``job_parameters`` to be used in combination with + ``notebook_params``, ``python_params``, ``jar_params``, ``spark_submit_params``, + ``python_named_params``, or ``dbt_commands``. Auto-forwarding is automatically skipped + when any of those parameter slots are used. .. code-block:: python diff --git a/providers/databricks/docs/operators/task.rst b/providers/databricks/docs/operators/task.rst index 5c446593531a0..b327aa20ccf97 100644 --- a/providers/databricks/docs/operators/task.rst +++ b/providers/databricks/docs/operators/task.rst @@ -44,3 +44,30 @@ Running a SQL query in Databricks using DatabricksTaskOperator :language: python :start-after: [START howto_operator_databricks_task_sql] :end-before: [END howto_operator_databricks_task_sql] + +Configuring Databricks-native task retries +------------------------------------------- + +Use ``max_retries``, ``min_retry_interval_millis`` and ``retry_on_timeout`` to configure +`Databricks-native task retries `_. +Databricks reruns failed task attempts within the same job run, so Airflow sees only the final result. +Set ``max_retries`` to ``-1`` to retry indefinitely, or ``0`` to disable retries. + +These settings are independent of the Airflow task-level ``retries`` parameter, which retries the +whole Airflow task. You can set the same fields directly in ``task_config``. When both are set, the +operator parameter takes precedence. If a field is unset, Databricks uses its default. + +Airflow ``retries`` behaves differently depending on where the operator runs. For a standalone +operator, each retry submits a new Databricks run. Inside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +the Airflow task monitors a sub-run that was already submitted by the workflow launch task, so a +retry only re-polls the terminal sub-run. Use ``max_retries`` to retry Databricks work inside a +workflow task group. + +Inside a +:class:`~airflow.providers.databricks.operators.databricks_workflow.DatabricksWorkflowTaskGroup`, +a task that exhausts a finite ``max_retries`` is reported as failed as soon as its final failed +attempt is observed, so downstream failure handling is not delayed by long-running sibling tasks. +Only unlimited retries (``max_retries=-1``) keep the Airflow task waiting (or deferring) until the +parent workflow run reaches a terminal state, because Databricks may still launch a retry attempt +under the same ``task_key`` until then. Sibling tasks in the run continue independently. diff --git a/providers/databricks/pyproject.toml b/providers/databricks/pyproject.toml index 1c2a58f95ac24..36f94b8cd6eee 100644 --- a/providers/databricks/pyproject.toml +++ b/providers/databricks/pyproject.toml @@ -66,18 +66,9 @@ dependencies = [ "databricks-sql-connector>=4.4.0", "aiohttp>=3.14.0, <4", "mergedeep>=1.3.4", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "pyarrow>=16.1.0; python_version < '3.13'", "pyarrow>=18.0.0; python_version >= '3.13' and python_version < '3.14'", "pyarrow>=22.0.0; python_version >= '3.14'", diff --git a/providers/databricks/src/airflow/providers/databricks/operators/databricks.py b/providers/databricks/src/airflow/providers/databricks/operators/databricks.py index 2d17a21587f2a..d820b099c8d3a 100644 --- a/providers/databricks/src/airflow/providers/databricks/operators/databricks.py +++ b/providers/databricks/src/airflow/providers/databricks/operators/databricks.py @@ -303,6 +303,16 @@ def _handle_deferrable_databricks_operator_completion(event: dict, log: Logger) "run_job_task": "job_parameters", } +# Parameter slots in run-now payload that Databricks API rejects when combined with job_parameters +_RUN_NOW_PARAM_SLOTS_CONFLICTING_WITH_JOB_PARAMETERS = ( + "notebook_params", + "python_params", + "jar_params", + "spark_submit_params", + "python_named_params", + "dbt_commands", +) + def _inject_airflow_params_into_task(task: dict, params: dict) -> None: """Set dict-shaped per-task parameter fields from ``params`` if they are not already set.""" @@ -1173,12 +1183,18 @@ class DatabricksRunNowOperator(ResumableJobMixin, BaseOperator): before polling begins so that a worker crash and retry reconnects to the existing run instead of triggering a duplicate run of the same job. Set to ``False`` to always trigger a fresh run on retry. Requires Airflow 3.3+; on earlier versions it is silently ignored. + :param forward_dag_params: Whether to forward Dag-level params as ``job_parameters`` + when no ``job_parameters`` are specified. (default: ``True``) .. note:: If ``job_parameters`` is not set in ``json`` and the operator's ``params`` dict is non-empty, the operator's ``params`` are automatically forwarded as ``job_parameters`` so that Airflow Dag params can be passed dynamically to Databricks runs without - hardcoding them in ``json``. + hardcoding them in ``json``. Set ``forward_dag_params=False`` to disable this. + Note that the Databricks API does not permit ``job_parameters`` to be used in combination + with ``notebook_params``, ``python_params``, ``jar_params``, ``spark_submit_params``, + ``python_named_params``, or ``dbt_commands``; auto-forwarding is automatically skipped + when any of those parameters are set. """ external_id_key = "databricks_run_now_id" @@ -1229,6 +1245,7 @@ def __init__( repair_run: bool = False, databricks_repair_reason_new_settings: dict[str, Any] | None = None, cancel_previous_runs: bool = False, + forward_dag_params: bool = True, **kwargs, ) -> None: """Create a new ``DatabricksRunNowOperator``.""" @@ -1254,6 +1271,7 @@ def __init__( self.repair_run = repair_run self.databricks_repair_reason_new_settings = databricks_repair_reason_new_settings or {} self.cancel_previous_runs = cancel_previous_runs + self.forward_dag_params = forward_dag_params # This variable will be used in case our task gets killed. self.run_id: int | None = None @@ -1321,7 +1339,12 @@ def _build_run_now_payload(self) -> dict[str, Any]: json["job_id"] = job_id del json["job_name"] - if not json.get("job_parameters") and self.params: + if ( + self.forward_dag_params + and not json.get("job_parameters") + and self.params + and not any(k in json for k in _RUN_NOW_PARAM_SLOTS_CONFLICTING_WITH_JOB_PARAMETERS) + ): json["job_parameters"] = dict(self.params) return json @@ -1650,6 +1673,9 @@ class DatabricksTaskBaseOperator(BaseOperator, ABC): :param wait_for_termination: if we should wait for termination of the job run. ``True`` by default. :param workflow_run_metadata: Metadata for the workflow run. This is used when the operator is used within a workflow. It is expected to be a dictionary containing the run_id and conn_id for the workflow. + :param max_retries: Databricks task retry count. Use ``-1`` for unlimited retries. + :param min_retry_interval_millis: Minimum interval between Databricks task retries. + :param retry_on_timeout: Whether Databricks retries timed-out tasks. """ def __init__( @@ -1667,6 +1693,9 @@ def __init__( polling_period_seconds: int = 5, wait_for_termination: bool = True, workflow_run_metadata: dict[str, Any] | None = None, + max_retries: int | str | None = None, + min_retry_interval_millis: int | str | None = None, + retry_on_timeout: bool | str | None = None, **kwargs: Any, ): self.caller = caller @@ -1682,6 +1711,9 @@ def __init__( self.polling_period_seconds = polling_period_seconds self.wait_for_termination = wait_for_termination self.workflow_run_metadata = workflow_run_metadata + self.max_retries = max_retries + self.min_retry_interval_millis = min_retry_interval_millis + self.retry_on_timeout = retry_on_timeout self.databricks_run_id: int | None = None @@ -1769,21 +1801,75 @@ def _get_task_base_json(self) -> dict[str, Any]: """Get the base json for the task.""" raise NotImplementedError() + def _retry_settings(self) -> dict[str, Any]: + """Databricks-native task retry settings that were explicitly provided.""" + settings: dict[str, Any] = {} + if self.max_retries is not None: + settings["max_retries"] = int(self.max_retries) + if self.min_retry_interval_millis is not None: + settings["min_retry_interval_millis"] = int(self.min_retry_interval_millis) + if self.retry_on_timeout is not None: + retry_on_timeout = self.retry_on_timeout + if isinstance(retry_on_timeout, str): + retry_on_timeout = retry_on_timeout.strip().lower() in ("true", "1", "yes") + settings["retry_on_timeout"] = bool(retry_on_timeout) + return settings + + def _has_retry_settings(self) -> bool: + """Whether any Databricks-native retry field is explicitly configured.""" + if self._retry_settings(): + return True + task_config = getattr(self, "task_config", {}) or {} + return any( + task_config.get(key) is not None + for key in ("max_retries", "min_retry_interval_millis", "retry_on_timeout") + ) + + def _resolved_max_retries(self) -> int | None: + """Return the resolved ``max_retries`` value, with operator args taking precedence.""" + task_config = getattr(self, "task_config", {}) or {} + max_retries = self.max_retries if self.max_retries is not None else task_config.get("max_retries") + + if isinstance(max_retries, bool) or max_retries is None: + return None + if isinstance(max_retries, str): + try: + return int(max_retries) + except ValueError: + self.log.warning( + "Ignoring unparsable max_retries value %r; treating as no retry limit configured.", + max_retries, + ) + return None + if not isinstance(max_retries, int): + return None + return max_retries + def _get_run_json(self) -> dict[str, Any]: """Get run json to be used for task submissions.""" - run_json = { - "run_name": self.databricks_task_key, - **self._get_task_base_json(), - } if self.new_cluster and self.existing_cluster_id: raise ValueError("Both new_cluster and existing_cluster_id are set. Only one should be set.") + cluster: dict[str, Any] if self.new_cluster: - run_json["new_cluster"] = self.new_cluster + cluster = {"new_cluster": self.new_cluster} elif self.existing_cluster_id: - run_json["existing_cluster_id"] = self.existing_cluster_id + cluster = {"existing_cluster_id": self.existing_cluster_id} else: raise ValueError("Must specify either existing_cluster_id or new_cluster.") - return run_json + + if not self._has_retry_settings(): + # No retry settings: keep the legacy single-task runs/submit shape unchanged. + return {"run_name": self.databricks_task_key, **self._get_task_base_json(), **cluster} + + # Retry settings are per-task SubmitTask fields, so submit the single task explicitly. + # The explicit task_key also gives monitoring a stable task to look up. + task = { + **self._get_task_base_json(), + "task_key": self.databricks_task_key, + **self._retry_settings(), + **cluster, + } + return {"run_name": self.databricks_task_key, "tasks": [task]} def _launch_job(self, context: Context | None = None) -> int | None: """Launch the job on Databricks.""" @@ -1834,13 +1920,14 @@ def _convert_to_databricks_workflow_task( base_task_json = self._get_task_base_json() result = { - "task_key": self.databricks_task_key, "depends_on": [ {"task_key": self._generate_databricks_task_key(task_id, task_dict)} for task_id in self.upstream_task_ids if task_id in relevant_upstreams ], **base_task_json, + "task_key": self.databricks_task_key, + **self._retry_settings(), } trigger_rule_value = ( @@ -1876,49 +1963,170 @@ def _convert_to_databricks_workflow_task( def monitor_databricks_job(self) -> None: """ - Monitor the Databricks job. - - Wait for the job to terminate. If deferrable, defer the task. + Monitor the Databricks job until it terminates and surface its result. + + Picks one of three monitoring strategies, depending on whether Databricks-native retry + attempts can occur and whether the operator runs inside a ``DatabricksWorkflowTaskGroup``: + + * standalone with native retries -> :meth:`_monitor_submit_run`, following the submit run + whose own terminal state already accounts for every retry attempt. + * workflow task with native retries -> :meth:`_monitor_workflow_task`, following the task's + latest attempt and tolerating in-flight retries until they are exhausted (or, for unlimited + retries, until the shared run is terminal). + * otherwise -> :meth:`_monitor_single_attempt`, following one attempt and reporting as soon + as it terminates (the historical behaviour, unchanged). """ if self.databricks_run_id is None: raise ValueError("Databricks job not yet launched. Please run launch_notebook_job first.") - current_task_run_id = self._get_current_databricks_task()["run_id"] - run = self._hook.get_run(current_task_run_id) - run_page_url = run["run_page_url"] - self.log.info("Check the task run in Databricks: %s", run_page_url) - run_state = RunState(**run["state"]) + + max_retries = self._resolved_max_retries() + if max_retries is None or (max_retries != -1 and max_retries <= 0): + self._monitor_single_attempt() + elif self._databricks_workflow_task_group is None: + self._monitor_submit_run() + else: + self._monitor_workflow_task() + + def _log_task_state(self, run_state: RunState) -> None: self.log.info( "Current state of the databricks task %s is %s", self.databricks_task_key, run_state.life_cycle_state, ) + + def _defer_on_run( + self, + run_id: int, + *, + workflow_run_id: int | None = None, + databricks_task_key: str | None = None, + max_retries: int | None = None, + ) -> None: + """Defer monitoring of ``run_id`` to the trigger, optionally with workflow-task context.""" + self.defer( + trigger=DatabricksExecutionTrigger( + run_id=run_id, + databricks_conn_id=self.databricks_conn_id, + polling_period_seconds=self.polling_period_seconds, + retry_limit=self.databricks_retry_limit, + retry_delay=self.databricks_retry_delay, + retry_args=self.databricks_retry_args, + caller=self.caller, + workflow_run_id=workflow_run_id, + databricks_task_key=databricks_task_key, + max_retries=max_retries, + ), + method_name=DEFER_METHOD_NAME, + ) + + def _monitor_single_attempt(self) -> None: + """Follow this task's attempt run and report as soon as it terminates (no native retries).""" + current_task_run_id = self._get_current_databricks_task()["run_id"] + run = self._hook.get_run(current_task_run_id) + self.log.info("Check the task run in Databricks: %s", run["run_page_url"]) + run_state = RunState(**run["state"]) + self._log_task_state(run_state) + if self.deferrable and not run_state.is_terminal: - self.defer( - trigger=DatabricksExecutionTrigger( - run_id=current_task_run_id, - databricks_conn_id=self.databricks_conn_id, - polling_period_seconds=self.polling_period_seconds, - retry_limit=self.databricks_retry_limit, - retry_delay=self.databricks_retry_delay, - retry_args=self.databricks_retry_args, - caller=self.caller, - ), - method_name=DEFER_METHOD_NAME, - ) + self._defer_on_run(current_task_run_id) + while not run_state.is_terminal: time.sleep(self.polling_period_seconds) run = self._hook.get_run(current_task_run_id) run_state = RunState(**run["state"]) + self._log_task_state(run_state) - self.log.info( - "Current state of the databricks task %s is %s", - self.databricks_task_key, - run_state.life_cycle_state, + errors = extract_failed_task_errors(self._hook, run, run_state) + self._handle_terminal_run_state(run_state, errors) + + def _monitor_workflow_task(self) -> None: + """ + Follow this task's latest attempt within a shared workflow run, tolerating native retries. + + Inside a ``DatabricksWorkflowTaskGroup`` the run holds sibling tasks, so the operator must + report when its own task finishes rather than wait for the whole run. A failed attempt is + final once finite retries are exhausted; unlimited retries fall back to the workflow run's + terminal state. Each poll re-resolves the latest attempt for the task key. + """ + workflow_run_id = self.databricks_run_id + if workflow_run_id is None: + raise ValueError("Databricks job not yet launched. Please run launch_notebook_job first.") + current_task = self._get_current_databricks_task() + current_task_run_id = current_task["run_id"] + run = self._hook.get_run(current_task_run_id) + self.log.info("Check the task run in Databricks: %s", run["run_page_url"]) + run_state = RunState(**run["state"]) + self._log_task_state(run_state) + attempt_number = current_task.get("attempt_number") + + # Defer whenever the outcome is not yet conclusive: a failed attempt with retries still + # available means a retry may follow, and the trigger waits for it without blocking a worker. + if self.deferrable and not self._workflow_task_is_conclusive( + run_state, workflow_run_id, attempt_number + ): + self._defer_on_run( + current_task_run_id, + workflow_run_id=workflow_run_id, + databricks_task_key=self.databricks_task_key, + max_retries=self._resolved_max_retries(), ) - # Extract errors from the run response using utility function + while not self._workflow_task_is_conclusive(run_state, workflow_run_id, attempt_number): + time.sleep(self.polling_period_seconds) + current_task = self._get_current_databricks_task() + current_task_run_id = current_task["run_id"] + run = self._hook.get_run(current_task_run_id) + run_state = RunState(**run["state"]) + self._log_task_state(run_state) + attempt_number = current_task.get("attempt_number") + errors = extract_failed_task_errors(self._hook, run, run_state) + self._handle_terminal_run_state(run_state, errors) + + def _workflow_task_is_conclusive( + self, run_state: RunState, workflow_run_id: int, attempt_number: int | None + ) -> bool: + """Whether the attempt is final: succeeded, retries exhausted, or the run is terminal.""" + if not run_state.is_terminal: + return False + if run_state.is_successful: + return True + max_retries = self._resolved_max_retries() + if ( + max_retries is not None + and max_retries != -1 + and attempt_number is not None + and attempt_number >= max_retries + ): + return True + parent_state = RunState(**self._hook.get_run(workflow_run_id)["state"]) + return parent_state.is_terminal + + def _monitor_submit_run(self) -> None: + """ + Wait for a standalone submit run to terminate, tolerating Databricks-native retries. + The submit run owns exactly this operator's task, so its own terminal state already + accounts for every native retry attempt — we follow the run rather than any single attempt. + """ + run_id = self.databricks_run_id + if run_id is None: + raise ValueError("Databricks job not yet launched. Please run launch_notebook_job first.") + run = self._hook.get_run(run_id) + self.log.info("Check the job run in Databricks: %s", run["run_page_url"]) + run_state = RunState(**run["state"]) + self.log.info("Current state of the databricks run %s is %s", run_id, run_state.life_cycle_state) + + if self.deferrable and not run_state.is_terminal: + self._defer_on_run(run_id) + + while not run_state.is_terminal: + time.sleep(self.polling_period_seconds) + run = self._hook.get_run(run_id) + run_state = RunState(**run["state"]) + self.log.info("Current state of the databricks run %s is %s", run_id, run_state.life_cycle_state) + + errors = extract_failed_task_errors(self._hook, run, run_state) self._handle_terminal_run_state(run_state, errors) def execute(self, context: Context) -> None: @@ -1950,6 +2158,31 @@ def execute_complete(self, context: dict | None, event: dict) -> None: errors = event.get("errors", []) self._handle_terminal_run_state(run_state, errors) + def on_kill(self) -> None: + if self.databricks_run_id is None: + return + if self._databricks_workflow_task_group: + # Workflow member: cancel only this task's child run, not the shared parent workflow run. + # Cancelling the parent would also stop all sibling tasks. + # If the child run_id cannot be resolved, log and bail out — do NOT fall back to the + # parent run_id as that would cancel sibling tasks. + try: + run_id_to_cancel = self._get_current_databricks_task()["run_id"] + except Exception: + self.log.exception( + "Task: %s could not resolve child run_id; skipping cancel to avoid stopping sibling tasks.", + self.task_id, + ) + return + else: + run_id_to_cancel = self.databricks_run_id + self._hook.cancel_run(run_id_to_cancel) + self.log.info( + "Task: %s with run_id: %s was requested to be cancelled.", + self.task_id, + run_id_to_cancel, + ) + class DatabricksNotebookOperator(DatabricksTaskBaseOperator): """ @@ -1985,11 +2218,17 @@ class DatabricksNotebookOperator(DatabricksTaskBaseOperator): :param wait_for_termination: if we should wait for termination of the job run. ``True`` by default. :param workflow_run_metadata: Metadata for the workflow run. This is used when the operator is used within a workflow. It is expected to be a dictionary containing the run_id and conn_id for the workflow. + :param max_retries: Databricks task retry count. Use ``-1`` for unlimited retries. + :param min_retry_interval_millis: Minimum interval between Databricks task retries. + :param retry_on_timeout: Whether Databricks retries timed-out tasks. """ template_fields = ( "notebook_params", "workflow_run_metadata", + "max_retries", + "min_retry_interval_millis", + "retry_on_timeout", ) CALLER = "DatabricksNotebookOperator" @@ -2010,6 +2249,9 @@ def __init__( polling_period_seconds: int = 5, wait_for_termination: bool = True, workflow_run_metadata: dict | None = None, + max_retries: int | str | None = None, + min_retry_interval_millis: int | str | None = None, + retry_on_timeout: bool | str | None = None, **kwargs: Any, ): self.notebook_path = notebook_path @@ -2030,6 +2272,9 @@ def __init__( polling_period_seconds=polling_period_seconds, wait_for_termination=wait_for_termination, workflow_run_metadata=workflow_run_metadata, + max_retries=max_retries, + min_retry_interval_millis=min_retry_interval_millis, + retry_on_timeout=retry_on_timeout, **kwargs, ) @@ -2128,6 +2373,9 @@ class DatabricksTaskOperator(DatabricksTaskBaseOperator): :param new_cluster: Specs for a new cluster on which this task will be run. :param polling_period_seconds: Controls the rate which we poll for the result of this notebook job run. :param wait_for_termination: if we should wait for termination of the job run. ``True`` by default. + :param max_retries: Databricks task retry count, overriding ``task_config`` when set. + :param min_retry_interval_millis: Minimum retry interval, overriding ``task_config`` when set. + :param retry_on_timeout: Whether Databricks retries timed-out tasks, overriding ``task_config`` when set. """ CALLER = "DatabricksTaskOperator" @@ -2135,6 +2383,9 @@ class DatabricksTaskOperator(DatabricksTaskBaseOperator): "databricks_conn_id", "task_config", "workflow_run_metadata", + "max_retries", + "min_retry_interval_millis", + "retry_on_timeout", ) def __init__( @@ -2151,6 +2402,9 @@ def __init__( polling_period_seconds: int = 5, wait_for_termination: bool = True, workflow_run_metadata: dict | None = None, + max_retries: int | str | None = None, + min_retry_interval_millis: int | str | None = None, + retry_on_timeout: bool | str | None = None, **kwargs, ): self.task_config = task_config @@ -2168,6 +2422,9 @@ def __init__( polling_period_seconds=polling_period_seconds, wait_for_termination=wait_for_termination, workflow_run_metadata=workflow_run_metadata, + max_retries=max_retries, + min_retry_interval_millis=min_retry_interval_millis, + retry_on_timeout=retry_on_timeout, **kwargs, ) diff --git a/providers/databricks/src/airflow/providers/databricks/triggers/databricks.py b/providers/databricks/src/airflow/providers/databricks/triggers/databricks.py index 57d489550f200..9706d2e4adfba 100644 --- a/providers/databricks/src/airflow/providers/databricks/triggers/databricks.py +++ b/providers/databricks/src/airflow/providers/databricks/triggers/databricks.py @@ -43,6 +43,9 @@ class DatabricksExecutionTrigger(BaseTrigger): :param run_page_url: The run page url. :param repair_run: Repair the databricks run in case of failure. :param caller: The name of the operator that is calling the hook. + :param workflow_run_id: Parent workflow run ID for task-level monitoring. + :param databricks_task_key: Task key to monitor within ``workflow_run_id``. + :param max_retries: Resolved Databricks-native ``max_retries`` for task-level monitoring. """ def __init__( @@ -56,6 +59,9 @@ def __init__( run_page_url: str | None = None, repair_run: bool = False, caller: str = "DatabricksExecutionTrigger", + workflow_run_id: int | None = None, + databricks_task_key: str | None = None, + max_retries: int | None = None, ) -> None: super().__init__() # Trigger kwargs cross Airflow's serialization boundary, so fail before storing invalid @@ -70,6 +76,9 @@ def __init__( self.run_page_url = run_page_url self.repair_run = repair_run self.caller = caller + self.workflow_run_id = workflow_run_id + self.databricks_task_key = databricks_task_key + self.max_retries = max_retries self.hook = DatabricksHook( databricks_conn_id, retry_limit=self.retry_limit, @@ -91,19 +100,40 @@ def serialize(self) -> tuple[str, dict[str, Any]]: "run_page_url": self.run_page_url, "repair_run": self.repair_run, "caller": self.caller, + "workflow_run_id": self.workflow_run_id, + "databricks_task_key": self.databricks_task_key, + "max_retries": self.max_retries, }, ) async def on_kill(self) -> None: """Cancel the Databricks run when the trigger is cancelled by a user action.""" - if self.run_id: - from asgiref.sync import sync_to_async + from asgiref.sync import sync_to_async + + run_id = self.run_id + if self.workflow_run_id is not None and self.databricks_task_key is not None: + # self.run_id may be an earlier, now-terminal attempt; cancel the task's latest attempt + # so a retry/repair launched under the same task_key is not left running. + tasks = await sync_to_async(self.hook.get_run_tasks)(self.workflow_run_id) + attempt = { + task["task_key"]: task for task in sorted(tasks, key=lambda task: task["start_time"]) + }.get(self.databricks_task_key) + if attempt: + run_id = attempt["run_id"] + if run_id: + self.log.info("Cancelling Databricks run %s.", run_id) + await sync_to_async(self.hook.cancel_run)(run_id) - self.log.info("Cancelling Databricks run %s.", self.run_id) - await sync_to_async(self.hook.cancel_run)(self.run_id) + def _monitors_workflow_task(self) -> bool: + """Whether this trigger follows one task inside a shared workflow run (see ``workflow_run_id``).""" + return bool(self.workflow_run_id and self.databricks_task_key) async def run(self): async with self.hook: + if self._monitors_workflow_task(): + async for event in self._run_workflow_task(): + yield event + return while True: run_state = await self.hook.a_get_run_state(self.run_id) if not run_state.is_terminal: @@ -129,6 +159,69 @@ async def run(self): ) return + async def _run_workflow_task(self): + """Monitor one task in a workflow run, tolerating in-flight retries/repairs.""" + from asgiref.sync import sync_to_async + + while True: + tasks = await sync_to_async(self.hook.get_run_tasks)(self.workflow_run_id) + sorted_task_runs = sorted(tasks, key=lambda task: task["start_time"]) + attempt = {task["task_key"]: task for task in sorted_task_runs}.get(self.databricks_task_key) + + if attempt is not None: + attempt_run_id = attempt["run_id"] + attempt_state = await self.hook.a_get_run_state(attempt_run_id) + + if attempt_state.is_terminal: + if attempt_state.is_successful: + yield TriggerEvent( + { + "run_id": attempt_run_id, + "run_page_url": self.run_page_url, + "run_state": attempt_state.to_json(), + "repair_run": self.repair_run, + "errors": [], + } + ) + return + # A failed attempt is final once finite retries are exhausted; otherwise wait + # for the parent run because another attempt may still appear. + attempt_number = attempt.get("attempt_number") + retries_exhausted = ( + self.max_retries is not None + and self.max_retries != -1 + and attempt_number is not None + and attempt_number >= self.max_retries + ) + if ( + retries_exhausted + or (await self.hook.a_get_run_state(self.workflow_run_id)).is_terminal + ): + run_info = await self.hook.a_get_run(attempt_run_id) + failed_tasks = await extract_failed_task_errors_async( + self.hook, run_info, attempt_state + ) + yield TriggerEvent( + { + "run_id": attempt_run_id, + "run_page_url": self.run_page_url, + "run_state": attempt_state.to_json(), + "repair_run": self.repair_run, + "errors": failed_tasks, + } + ) + return + + # attempt is None when the task has not yet surfaced in the run (e.g. just after launch); + # keep polling rather than crashing on a missing task_key. + self.log.info( + "databricks task %s not yet conclusive in run %s. sleeping for %s seconds", + self.databricks_task_key, + self.workflow_run_id, + self.polling_period_seconds, + ) + await asyncio.sleep(self.polling_period_seconds) + class DatabricksSQLStatementExecutionTrigger(BaseTrigger): """ diff --git a/providers/databricks/tests/unit/databricks/operators/test_databricks.py b/providers/databricks/tests/unit/databricks/operators/test_databricks.py index 9c83c19ebe915..0f190da3661d0 100644 --- a/providers/databricks/tests/unit/databricks/operators/test_databricks.py +++ b/providers/databricks/tests/unit/databricks/operators/test_databricks.py @@ -2028,7 +2028,7 @@ def test_execute_does_not_mutate_json_template_field(self, db_mock_class): durable=False, task_id=TASK_ID, job_id=JOB_ID, - json={"notebook_params": {"a": "b"}}, + json={"idempotency_token": "token_123"}, params={"env": "prod"}, ) op.render_template_fields(context={"ds": DATE}) @@ -2925,6 +2925,62 @@ def test_run_now_does_not_override_existing_job_parameters(self, db_mock_class): actual = db_mock.run_now.call_args.args[0] assert actual["job_parameters"] == {"explicit": "value"} + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_run_now_does_not_inject_airflow_params_when_forward_dag_params_is_false(self, db_mock_class): + """ + When ``forward_dag_params`` is False, the operator's ``params`` should + not be forwarded as ``job_parameters``. + """ + op = DatabricksRunNowOperator( + durable=False, + task_id=TASK_ID, + job_id=JOB_ID, + forward_dag_params=False, + params={"env": "prod", "batch_size": 100}, + ) + db_mock = db_mock_class.return_value + db_mock.run_now.return_value = RUN_ID + db_mock.get_run = make_run_with_state_mock("TERMINATED", "SUCCESS") + + op.execute(None) + + actual = db_mock.run_now.call_args.args[0] + assert "job_parameters" not in actual + + @pytest.mark.parametrize( + ("slot_param", "slot_val"), + [ + ("notebook_params", {"foo": "bar"}), + ("python_params", ["foo", "bar"]), + ("jar_params", ["foo", "bar"]), + ("spark_submit_params", ["--class", "Foo"]), + ("python_named_params", {"foo": "bar"}), + ("dbt_commands", ["dbt deps", "dbt run"]), + ], + ) + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_run_now_skips_param_injection_with_legacy_param_slots(self, db_mock_class, slot_param, slot_val): + """ + When any legacy param slot (notebook_params, etc.) is set, auto-injection of ``self.params`` + into ``job_parameters`` must be skipped because the Databricks API rejects combining them. + """ + op = DatabricksRunNowOperator( + durable=False, + task_id=TASK_ID, + job_id=JOB_ID, + params={"env": "prod"}, + **{slot_param: slot_val}, + ) + db_mock = db_mock_class.return_value + db_mock.run_now.return_value = RUN_ID + db_mock.get_run = make_run_with_state_mock("TERMINATED", "SUCCESS") + + op.execute(None) + + actual = db_mock.run_now.call_args.args[0] + assert slot_param in actual + assert "job_parameters" not in actual + @pytest.mark.skipif( not AIRFLOW_V_3_3_PLUS, reason="task_state_store (durable execution) requires Airflow 3.3+" @@ -3631,6 +3687,103 @@ def test_execute_with_deferrable(self, mock_get_current_task, mock_databricks_ho "Trigger is not a DatabricksExecutionTrigger" ) assert exec_info.value.method_name == "execute_complete" + # Without native retries configured, the trigger keeps its original parent-unaware behavior. + assert exec_info.value.trigger.workflow_run_id is None + assert exec_info.value.trigger.databricks_task_key is None + + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_execute_standalone_with_retries_defers_on_submit_run(self, mock_databricks_hook): + # A standalone operator with native retries follows its own submit run to a terminal state, + # so the defer targets that run directly (no per-task workflow context). + mock_databricks_hook.return_value.get_run.return_value = { + "state": {"life_cycle_state": "PENDING"}, + "run_page_url": "test_url", + } + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + wait_for_termination=True, + deferrable=True, + max_retries=2, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(TaskDeferred) as exec_info: + operator.monitor_databricks_job() + assert exec_info.value.trigger.run_id == 12345 + assert exec_info.value.trigger.workflow_run_id is None + assert exec_info.value.trigger.databricks_task_key is None + + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + ) + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_execute_with_deferrable_passes_workflow_context_when_retries_configured( + self, mock_get_current_task, mock_databricks_hook, mock_workflow_tg + ): + # Inside a workflow task group the run is shared, so the defer carries the workflow context + # and the trigger tracks this task's own attempt within that run. + mock_workflow_tg.return_value = MagicMock() + mock_get_current_task.return_value = {"run_id": "attempt-1"} + mock_databricks_hook.return_value.get_run.return_value = { + "state": {"life_cycle_state": "PENDING"}, + "run_page_url": "test_url", + } + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + wait_for_termination=True, + deferrable=True, + max_retries=2, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(TaskDeferred) as exec_info: + operator.monitor_databricks_job() + assert exec_info.value.trigger.workflow_run_id == 12345 + assert exec_info.value.trigger.databricks_task_key == operator.databricks_task_key + + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + ) + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_execute_with_deferrable_and_max_retries_zero_keeps_single_attempt_monitoring( + self, mock_get_current_task, mock_databricks_hook, mock_workflow_tg + ): + mock_workflow_tg.return_value = MagicMock() + mock_get_current_task.return_value = {"run_id": "attempt-1"} + mock_databricks_hook.return_value.get_run.return_value = { + "state": {"life_cycle_state": "PENDING"}, + "run_page_url": "test_url", + } + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + wait_for_termination=True, + deferrable=True, + max_retries=0, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(TaskDeferred) as exec_info: + operator.monitor_databricks_job() + assert exec_info.value.trigger.run_id == "attempt-1" + assert exec_info.value.trigger.workflow_run_id is None + assert exec_info.value.trigger.databricks_task_key is None @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") @mock.patch( @@ -3706,6 +3859,279 @@ def test_monitor_databricks_job_failed(self, mock_get_current_task, mock_databri exception_message = "Task failed. Final state FAILED. Reason: FAILURE. Errors: []" assert exception_message == str(exc_info.value) + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + ) + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_monitor_databricks_job_retry_in_flight_succeeds( + self, mock_get_current_task, mock_databricks_hook, mock_sleep, mock_workflow_tg + ): + # Inside a workflow task group the run is shared, so the task tracks its own attempt: the + # first attempt fails while the workflow run is still active; a retried attempt then + # succeeds and the Airflow task must not fail. + mock_workflow_tg.return_value = MagicMock() + mock_get_current_task.side_effect = [{"run_id": "attempt-1"}, {"run_id": "attempt-2"}] + runs = { + "attempt-1": { + "state": { + "life_cycle_state": "TERMINATED", + "result_state": "FAILED", + "state_message": "first attempt failed", + }, + "run_page_url": "url-1", + }, + 12345: {"state": {"life_cycle_state": "RUNNING"}, "run_page_url": "parent"}, + "attempt-2": { + "state": {"life_cycle_state": "TERMINATED", "result_state": "SUCCESS"}, + "run_page_url": "url-2", + }, + } + mock_databricks_hook.return_value.get_run.side_effect = lambda run_id: runs[run_id] + + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + max_retries=1, + ) + operator.databricks_run_id = 12345 + + operator.monitor_databricks_job() + mock_sleep.assert_called_once() + + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_monitor_standalone_submit_run_follows_run_to_terminal_state( + self, mock_databricks_hook, mock_sleep + ): + # A standalone submit run stays active while Databricks retries the task; the operator + # follows the run (not an attempt) and reports only once the run itself terminates. + run_states = iter( + [ + {"life_cycle_state": "RUNNING"}, + {"life_cycle_state": "TERMINATED", "result_state": "SUCCESS"}, + ] + ) + mock_databricks_hook.return_value.get_run.side_effect = lambda run_id: { + "state": next(run_states), + "run_page_url": "url", + } + + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + max_retries=1, + ) + operator.databricks_run_id = 12345 + + operator.monitor_databricks_job() + # Only the submit run is polled; the per-attempt resolver is never used. + mock_databricks_hook.return_value.get_run.assert_called_with(12345) + mock_databricks_hook.return_value.get_run_tasks.assert_not_called() + + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_monitor_databricks_job_without_retries_fails_immediately( + self, mock_get_current_task, mock_databricks_hook, mock_sleep + ): + # Without native retries configured the task fails as soon as its own attempt terminates, + # even if the parent run is still active. The parent run state must not be polled. + mock_get_current_task.return_value = {"run_id": "attempt-1"} + mock_databricks_hook.return_value.get_run.return_value = { + "state": { + "life_cycle_state": "TERMINATED", + "result_state": "FAILED", + "state_message": "attempt failed", + }, + "run_page_url": "url", + } + + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + ) + operator.databricks_run_id = 12345 + + with pytest.raises(AirflowException): + operator.monitor_databricks_job() + mock_sleep.assert_not_called() + # Only the attempt run is polled (once, before the loop); the parent run is never fetched. + mock_databricks_hook.return_value.get_run.assert_called_once_with("attempt-1") + + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + ) + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_monitor_workflow_task_reports_failure_once_retries_exhausted( + self, mock_get_current_task, mock_databricks_hook, mock_sleep, mock_workflow_tg + ): + mock_workflow_tg.return_value = MagicMock() + mock_get_current_task.return_value = {"run_id": "attempt-2", "attempt_number": 1} + mock_databricks_hook.return_value.get_run.return_value = { + "state": { + "life_cycle_state": "TERMINATED", + "result_state": "FAILED", + "state_message": "final attempt failed", + }, + "run_page_url": "url", + } + + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + max_retries=1, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(AirflowException): + operator.monitor_databricks_job() + mock_sleep.assert_not_called() + mock_databricks_hook.return_value.get_run.assert_called_once_with("attempt-2") + + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + ) + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksNotebookOperator._get_current_databricks_task" + ) + def test_monitor_workflow_task_unlimited_retries_waits_for_parent( + self, mock_get_current_task, mock_databricks_hook, mock_sleep, mock_workflow_tg + ): + mock_workflow_tg.return_value = MagicMock() + mock_get_current_task.return_value = {"run_id": "attempt-1", "attempt_number": 5} + parent_states = iter( + [ + {"life_cycle_state": "RUNNING"}, + {"life_cycle_state": "TERMINATED", "result_state": "FAILED", "state_message": "failed"}, + ] + ) + + def fake_get_run(run_id): + if run_id == 12345: + return {"state": next(parent_states), "run_page_url": "parent"} + return { + "state": { + "life_cycle_state": "TERMINATED", + "result_state": "FAILED", + "state_message": "failed", + }, + "run_page_url": "url", + } + + mock_databricks_hook.return_value.get_run.side_effect = fake_get_run + + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + max_retries=-1, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(AirflowException): + operator.monitor_databricks_job() + mock_sleep.assert_called_once() + mock_databricks_hook.return_value.get_run.assert_any_call(12345) + + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_reshaped_submit_run_without_native_retries_resolves_task_for_monitoring( + self, mock_databricks_hook, mock_sleep + ): + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + min_retry_interval_millis=2000, + ) + run_json = operator._get_run_json() + assert "tasks" in run_json + assert run_json["tasks"][0]["task_key"] == operator.databricks_task_key + assert operator._resolved_max_retries() is None + + operator.databricks_run_id = 12345 + mock_databricks_hook.return_value.get_run_tasks.return_value = [ + {"task_key": operator.databricks_task_key, "run_id": "attempt-1", "start_time": 1} + ] + mock_databricks_hook.return_value.get_run.return_value = { + "state": {"life_cycle_state": "TERMINATED", "result_state": "SUCCESS"}, + "run_page_url": "url", + } + + operator.monitor_databricks_job() + mock_databricks_hook.return_value.get_run_tasks.assert_called_once_with(12345) + mock_databricks_hook.return_value.get_run.assert_called_once_with("attempt-1") + + @pytest.mark.parametrize( + "operator", + [ + DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="WORKSPACE", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + max_retries="{{ params.max_retries }}", + min_retry_interval_millis="{{ params.min_retry_interval_millis }}", + retry_on_timeout="{{ params.retry_on_timeout }}", + ), + DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + task_config={}, + max_retries="{{ params.max_retries }}", + min_retry_interval_millis="{{ params.min_retry_interval_millis }}", + retry_on_timeout="{{ params.retry_on_timeout }}", + ), + ], + ) + def test_retry_params_render_from_templates(self, operator): + """Retry fields are templatable: Jinja values render and coerce to typed retry settings. + + Guards the ``template_fields`` membership on both subclasses (which keep separate tuples): + an un-templated field would keep its literal ``{{ ... }}`` string and fail coercion. + """ + operator.render_template_fields( + context={ + "params": { + "max_retries": 3, + "min_retry_interval_millis": 2000, + "retry_on_timeout": "true", + } + } + ) + assert operator._retry_settings() == { + "max_retries": 3, + "min_retry_interval_millis": 2000, + "retry_on_timeout": True, + } + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") def test_launch_notebook_job(self, mock_databricks_hook): operator = DatabricksNotebookOperator( @@ -3858,6 +4284,88 @@ def test_convert_to_databricks_workflow_task(self): assert task_json == expected_json + @pytest.mark.parametrize( + ("retry_kwargs", "expected"), + [ + ( + {"max_retries": -1, "min_retry_interval_millis": 2000, "retry_on_timeout": True}, + {"max_retries": -1, "min_retry_interval_millis": 2000, "retry_on_timeout": True}, + ), + ({"max_retries": 0}, {"max_retries": 0}), + ({"retry_on_timeout": False}, {"retry_on_timeout": False}), + ], + ) + def test_get_run_json_retry_settings(self, retry_kwargs, expected): + """Retry settings are added to the submitted task only when explicitly provided. + + They must live inside ``tasks[0]`` (a Databricks ``SubmitTask``); at the top level of a + runs/submit payload Databricks silently ignores them. + """ + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + **retry_kwargs, + ) + run_json = operator._get_run_json() + assert "tasks" in run_json + task = run_json["tasks"][0] + assert task["task_key"] == operator.databricks_task_key + assert task["existing_cluster_id"] == "existing_cluster_id" + assert task["notebook_task"]["notebook_path"] == "test_path" + assert "existing_cluster_id" not in run_json + for key in ("max_retries", "min_retry_interval_millis", "retry_on_timeout"): + assert key not in run_json + if key in expected: + assert task[key] == expected[key] + else: + assert key not in task + + def test_get_run_json_without_retries_uses_legacy_top_level_shape(self): + """Without native retries the payload is the legacy single-task runs/submit shape (unchanged).""" + operator = DatabricksNotebookOperator( + task_id="test_task", + notebook_path="test_path", + source="test_source", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + ) + run_json = operator._get_run_json() + assert run_json["run_name"] == operator.databricks_task_key + assert run_json["existing_cluster_id"] == "existing_cluster_id" + assert run_json["notebook_task"]["notebook_path"] == "test_path" + # No reshape into the multi-task form when retries are not configured. + assert "tasks" not in run_json + + def test_convert_to_databricks_workflow_task_includes_retry_settings(self): + """Retry settings provided to the operator are included in the workflow task JSON.""" + dag = DAG(dag_id="example_dag", schedule=None, start_date=DEFAULT_DATE) + operator = DatabricksNotebookOperator( + notebook_path="/path/to/notebook", + source="WORKSPACE", + task_id="test_task", + max_retries=2, + min_retry_interval_millis=1000, + retry_on_timeout=True, + dag=dag, + ) + + databricks_workflow_task_group = MagicMock() + databricks_workflow_task_group.notebook_packages = [] + databricks_workflow_task_group.notebook_params = {} + + operator.task_group = databricks_workflow_task_group + relevant_upstreams = [] + task_dict = {} + + task_json = operator._convert_to_databricks_workflow_task(relevant_upstreams, task_dict) + + assert task_json["max_retries"] == 2 + assert task_json["min_retry_interval_millis"] == 1000 + assert task_json["retry_on_timeout"] is True + @pytest.mark.parametrize( ("trigger_rule", "expected_run_if"), [ @@ -3998,6 +4506,69 @@ def test_get_task_base_json(self): assert operator.task_config == task_config assert task_base_json == task_config + def test_get_run_json_operator_task_key_wins_over_task_config(self): + """A task_key in task_config must not shadow the operator-managed key used for monitoring. + + The operator only injects its own task_key in the reshaped (native-retry) payload, where + monitoring re-resolves the task by that key, so the guarantee is exercised with retries set. + """ + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + max_retries=2, + task_config={"task_key": "user_supplied_key", "notebook_task": {"notebook_path": "/p"}}, + ) + task = operator._get_run_json()["tasks"][0] + assert task["task_key"] == operator.databricks_task_key + + def test_convert_to_databricks_workflow_task_includes_task_config_retry_settings(self): + """Retry settings supplied via task_config surface in the workflow task JSON.""" + dag = DAG(dag_id="example_dag", schedule=None, start_date=DEFAULT_DATE) + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + task_config={ + "notebook_task": {"notebook_path": "/path"}, + "max_retries": 5, + "min_retry_interval_millis": 1000, + "retry_on_timeout": True, + }, + dag=dag, + ) + operator.task_group = MagicMock() + + task_json = operator._convert_to_databricks_workflow_task([], {}) + + assert task_json["max_retries"] == 5 + assert task_json["min_retry_interval_millis"] == 1000 + assert task_json["retry_on_timeout"] is True + + def test_convert_to_databricks_workflow_task_operator_retry_overrides_task_config(self): + """An operator-level retry value overrides the task_config value in the workflow task JSON.""" + dag = DAG(dag_id="example_dag", schedule=None, start_date=DEFAULT_DATE) + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + task_config={ + "notebook_task": {"notebook_path": "/path"}, + "max_retries": 5, + "min_retry_interval_millis": 1000, + "retry_on_timeout": False, + }, + max_retries=2, + min_retry_interval_millis=2000, + retry_on_timeout=True, + dag=dag, + ) + operator.task_group = MagicMock() + + task_json = operator._convert_to_databricks_workflow_task([], {}) + + assert task_json["max_retries"] == 2 + assert task_json["min_retry_interval_millis"] == 2000 + assert task_json["retry_on_timeout"] is True + def test_generate_databricks_task_key(self): task_config = {} operator = DatabricksTaskOperator( @@ -4034,3 +4605,198 @@ def test_user_databricks_task_key(self): expected_task_key = "test_task_key" assert expected_task_key == operator.databricks_task_key + + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_on_kill_cancels_run(self, db_mock_class): + operator = DatabricksTaskOperator( + task_id="task", + task_config={"sql_task": {"query": {"query_id": "abc"}}}, + ) + db_mock = db_mock_class.return_value + operator.databricks_run_id = 1 + operator.on_kill() + db_mock.cancel_run.assert_called_once_with(1) + + def test_on_kill_does_nothing_when_run_id_is_none(self): + operator = DatabricksTaskOperator( + task_id="task", + task_config={"sql_task": {"query": {"query_id": "abc"}}}, + ) + with mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") as db_mock_class: + operator.on_kill() + db_mock_class.return_value.cancel_run.assert_not_called() + + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_on_kill_workflow_member_cancels_child_run(self, db_mock_class): + operator = DatabricksTaskOperator( + task_id="task", + task_config={"sql_task": {"query": {"query_id": "abc"}}}, + ) + db_mock = db_mock_class.return_value + operator.databricks_run_id = 1 + with mock.patch.object( + operator, + "_get_current_databricks_task", + return_value={"run_id": 999, "task_key": "task"}, + ): + with mock.patch( + "airflow.providers.databricks.operators.databricks" + ".DatabricksTaskBaseOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + return_value=object(), + ): + operator.on_kill() + db_mock.cancel_run.assert_called_once_with(999) + + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_on_kill_workflow_member_get_task_raises_does_not_cancel_parent(self, db_mock_class): + operator = DatabricksTaskOperator( + task_id="task", + task_config={"sql_task": {"query": {"query_id": "abc"}}}, + ) + db_mock = db_mock_class.return_value + operator.databricks_run_id = 1 + with mock.patch.object( + operator, + "_get_current_databricks_task", + side_effect=Exception("API error"), + ): + with mock.patch( + "airflow.providers.databricks.operators.databricks" + ".DatabricksTaskBaseOperator._databricks_workflow_task_group", + new_callable=mock.PropertyMock, + return_value=object(), + ): + operator.on_kill() + db_mock.cancel_run.assert_not_called() + + @pytest.mark.parametrize( + ("field", "task_config_retries", "retry_kwargs", "expected"), + [ + ("max_retries", {"max_retries": 1}, {}, 1), + ("max_retries", {"max_retries": 1}, {"max_retries": 0}, 0), + ( + "min_retry_interval_millis", + {"min_retry_interval_millis": 1000}, + {"min_retry_interval_millis": 5000}, + 5000, + ), + ("retry_on_timeout", {"retry_on_timeout": True}, {"retry_on_timeout": False}, False), + ], + ) + def test_get_run_json_retry_settings(self, field, task_config_retries, retry_kwargs, expected): + """Operator-level retry settings are applied and take precedence over task_config values.""" + task_config = { + "notebook_task": {"notebook_path": "/path", "source": "WORKSPACE", "base_parameters": {}}, + **task_config_retries, + } + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + task_config=task_config, + **retry_kwargs, + ) + run_json = operator._get_run_json() + assert run_json["tasks"][0][field] == expected + assert field not in run_json + + @pytest.mark.parametrize("max_retries", [2, "2"]) + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_monitor_uses_retry_strategy_when_retries_set_in_task_config( + self, mock_databricks_hook, mock_sleep, max_retries + ): + # max_retries supplied only through task_config must still select retry-aware monitoring + # (follow the submit run), not the single-attempt path. + run_states = iter( + [ + {"life_cycle_state": "RUNNING"}, + {"life_cycle_state": "TERMINATED", "result_state": "SUCCESS"}, + ] + ) + mock_databricks_hook.return_value.get_run.side_effect = lambda run_id: { + "state": next(run_states), + "run_page_url": "url", + } + + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + task_config={ + "notebook_task": {"notebook_path": "/path", "source": "WORKSPACE", "base_parameters": {}}, + "max_retries": max_retries, + }, + ) + operator.databricks_run_id = 12345 + + operator.monitor_databricks_job() + mock_databricks_hook.return_value.get_run.assert_called_with(12345) + mock_databricks_hook.return_value.get_run_tasks.assert_not_called() + + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + @mock.patch( + "airflow.providers.databricks.operators.databricks.DatabricksTaskOperator._get_current_databricks_task" + ) + def test_monitor_max_retries_zero_operator_arg_overrides_task_config_retries( + self, mock_get_current_task, mock_databricks_hook, mock_sleep + ): + mock_get_current_task.return_value = {"run_id": "attempt-1"} + mock_databricks_hook.return_value.get_run.return_value = { + "state": { + "life_cycle_state": "TERMINATED", + "result_state": "FAILED", + "state_message": "attempt failed", + }, + "run_page_url": "url", + } + + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + task_config={ + "notebook_task": {"notebook_path": "/path", "source": "WORKSPACE", "base_parameters": {}}, + "max_retries": 2, + }, + max_retries=0, + ) + operator.databricks_run_id = 12345 + + with pytest.raises(AirflowException): + operator.monitor_databricks_job() + mock_sleep.assert_not_called() + mock_databricks_hook.return_value.get_run.assert_called_once_with("attempt-1") + + @mock.patch("airflow.providers.databricks.operators.databricks.time.sleep") + @mock.patch("airflow.providers.databricks.operators.databricks.DatabricksHook") + def test_monitor_waits_out_waiting_for_retry_state(self, mock_databricks_hook, mock_sleep): + # A native retry surfaces WAITING_FOR_RETRY between attempts; the poll must keep waiting + # rather than crash on an unexpected life cycle state. + run_states = iter( + [ + {"life_cycle_state": "RUNNING"}, + {"life_cycle_state": "WAITING_FOR_RETRY"}, + {"life_cycle_state": "TERMINATED", "result_state": "SUCCESS"}, + ] + ) + mock_databricks_hook.return_value.get_run.side_effect = lambda run_id: { + "state": next(run_states), + "run_page_url": "url", + } + + operator = DatabricksTaskOperator( + task_id="test_task", + databricks_conn_id="test_conn_id", + existing_cluster_id="existing_cluster_id", + task_config={ + "notebook_task": {"notebook_path": "/path", "source": "WORKSPACE", "base_parameters": {}}, + "max_retries": 2, + }, + ) + operator.databricks_run_id = 12345 + + operator.monitor_databricks_job() + assert mock_databricks_hook.return_value.get_run.call_count == 3 diff --git a/providers/databricks/tests/unit/databricks/triggers/test_databricks.py b/providers/databricks/tests/unit/databricks/triggers/test_databricks.py index 8854eb03fb5bc..ad204297dc80e 100644 --- a/providers/databricks/tests/unit/databricks/triggers/test_databricks.py +++ b/providers/databricks/tests/unit/databricks/triggers/test_databricks.py @@ -189,6 +189,9 @@ def test_serialize(self): "run_page_url": RUN_PAGE_URL, "repair_run": False, "caller": "DatabricksExecutionTrigger", + "workflow_run_id": None, + "databricks_task_key": None, + "max_retries": None, }, ) @@ -306,12 +309,210 @@ async def test_sleep_between_retries( mock_sleep.assert_called_once() mock_sleep.assert_called_with(POLLING_INTERVAL_SECONDS) + @pytest.mark.asyncio + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_output") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run") + @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state") + async def test_run_waits_out_waiting_for_retry_state( + self, mock_get_run_state, mock_sleep, mock_get_run, mock_get_run_output + ): + # A native retry surfaces WAITING_FOR_RETRY between attempts; the trigger must keep polling + # rather than crash on an unexpected life cycle state. + mock_get_run_state.side_effect = [ + RunState(life_cycle_state="WAITING_FOR_RETRY", state_message="", result_state=""), + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"), + ] + mock_get_run.return_value = GET_RUN_RESPONSE_TERMINATED + mock_get_run_output.return_value = GET_RUN_OUTPUT_RESPONSE + + async for event in self.trigger.run(): + assert event == TriggerEvent( + { + "run_id": RUN_ID, + "run_state": RunState( + life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS" + ).to_json(), + "run_page_url": RUN_PAGE_URL, + "repair_run": False, + "errors": [], + } + ) + mock_sleep.assert_called_once_with(POLLING_INTERVAL_SECONDS) + + @pytest.mark.asyncio + @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.get_run_tasks") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state") + async def test_run_workflow_task_retry_in_flight_succeeds( + self, mock_get_run_state, mock_get_run_tasks, mock_sleep + ): + # First attempt fails while the parent run is still active; a later attempt succeeds and + # the trigger must emit a success event rather than failing on the first attempt. + mock_get_run_tasks.side_effect = [ + [{"run_id": TASK_RUN_ID1, "task_key": TASK_RUN_ID1_KEY, "start_time": 1}], + [ + {"run_id": TASK_RUN_ID1, "task_key": TASK_RUN_ID1_KEY, "start_time": 1}, + {"run_id": TASK_RUN_ID2, "task_key": TASK_RUN_ID1_KEY, "start_time": 2}, + ], + ] + mock_get_run_state.side_effect = [ + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED"), + RunState(life_cycle_state=LIFE_CYCLE_STATE_PENDING, state_message="", result_state=""), + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS"), + ] + + trigger = DatabricksExecutionTrigger( + run_id=TASK_RUN_ID1, + databricks_conn_id=DEFAULT_CONN_ID, + polling_period_seconds=POLLING_INTERVAL_SECONDS, + run_page_url=RUN_PAGE_URL, + workflow_run_id=RUN_ID, + databricks_task_key=TASK_RUN_ID1_KEY, + ) + + events = [event async for event in trigger.run()] + assert events == [ + TriggerEvent( + { + "run_id": TASK_RUN_ID2, + "run_page_url": RUN_PAGE_URL, + "run_state": RunState( + life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="SUCCESS" + ).to_json(), + "repair_run": False, + "errors": [], + } + ) + ] + mock_sleep.assert_called_once_with(POLLING_INTERVAL_SECONDS) + + @pytest.mark.asyncio + @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_output") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.get_run_tasks") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state") + async def test_run_workflow_task_failed_attempt_waits_for_parent( + self, mock_get_run_state, mock_get_run_tasks, mock_get_run, mock_get_run_output, mock_sleep + ): + mock_get_run_tasks.return_value = [ + {"run_id": TASK_RUN_ID1, "task_key": TASK_RUN_ID1_KEY, "start_time": 1} + ] + mock_get_run_state.side_effect = [ + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED"), + RunState(life_cycle_state="RUNNING", state_message="", result_state=""), + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED"), + RunState(life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED"), + ] + mock_get_run.return_value = GET_RUN_RESPONSE_TERMINATED_WITH_FAILED + mock_get_run_output.return_value = GET_RUN_OUTPUT_RESPONSE + + trigger = DatabricksExecutionTrigger( + run_id=TASK_RUN_ID1, + databricks_conn_id=DEFAULT_CONN_ID, + polling_period_seconds=POLLING_INTERVAL_SECONDS, + run_page_url=RUN_PAGE_URL, + workflow_run_id=RUN_ID, + databricks_task_key=TASK_RUN_ID1_KEY, + ) + + events = [event async for event in trigger.run()] + assert events == [ + TriggerEvent( + { + "run_id": TASK_RUN_ID1, + "run_page_url": RUN_PAGE_URL, + "run_state": RunState( + life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED" + ).to_json(), + "repair_run": False, + "errors": [ + {"task_key": TASK_RUN_ID1_KEY, "run_id": TASK_RUN_ID1, "error": ERROR_MESSAGE}, + {"task_key": TASK_RUN_ID3_KEY, "run_id": TASK_RUN_ID3, "error": ERROR_MESSAGE}, + ], + } + ) + ] + mock_sleep.assert_called_once_with(POLLING_INTERVAL_SECONDS) + + @pytest.mark.asyncio + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_output") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run") + @mock.patch("airflow.providers.databricks.triggers.databricks.asyncio.sleep") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.get_run_tasks") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.a_get_run_state") + async def test_run_workflow_task_reports_failure_once_retries_exhausted( + self, mock_get_run_state, mock_get_run_tasks, mock_sleep, mock_get_run, mock_get_run_output + ): + mock_get_run_tasks.return_value = [ + {"run_id": TASK_RUN_ID1, "task_key": TASK_RUN_ID1_KEY, "start_time": 1, "attempt_number": 1} + ] + mock_get_run_state.return_value = RunState( + life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED" + ) + mock_get_run.return_value = GET_RUN_RESPONSE_TERMINATED_WITH_FAILED + mock_get_run_output.return_value = GET_RUN_OUTPUT_RESPONSE + + trigger = DatabricksExecutionTrigger( + run_id=TASK_RUN_ID1, + databricks_conn_id=DEFAULT_CONN_ID, + polling_period_seconds=POLLING_INTERVAL_SECONDS, + run_page_url=RUN_PAGE_URL, + workflow_run_id=RUN_ID, + databricks_task_key=TASK_RUN_ID1_KEY, + max_retries=1, + ) + + events = [event async for event in trigger.run()] + assert events == [ + TriggerEvent( + { + "run_id": TASK_RUN_ID1, + "run_page_url": RUN_PAGE_URL, + "run_state": RunState( + life_cycle_state=LIFE_CYCLE_STATE_TERMINATED, state_message="", result_state="FAILED" + ).to_json(), + "repair_run": False, + "errors": [ + {"task_key": TASK_RUN_ID1_KEY, "run_id": TASK_RUN_ID1, "error": ERROR_MESSAGE}, + {"task_key": TASK_RUN_ID3_KEY, "run_id": TASK_RUN_ID3, "error": ERROR_MESSAGE}, + ], + } + ) + ] + mock_sleep.assert_not_called() + mock_get_run_state.assert_called_once_with(TASK_RUN_ID1) + @pytest.mark.asyncio @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.cancel_run") async def test_on_kill_cancels_run(self, mock_cancel_run): await self.trigger.on_kill() mock_cancel_run.assert_called_once_with(RUN_ID) + @pytest.mark.asyncio + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.get_run_tasks") + @mock.patch("airflow.providers.databricks.hooks.databricks.DatabricksHook.cancel_run") + async def test_on_kill_workflow_task_cancels_latest_attempt(self, mock_cancel_run, mock_get_run_tasks): + # The original attempt may be terminal after a retry; on_kill must cancel the latest attempt + # (under the same task_key) instead of the now-stale run_id the trigger was created with. + mock_get_run_tasks.return_value = [ + {"run_id": TASK_RUN_ID1, "task_key": TASK_RUN_ID1_KEY, "start_time": 1}, + {"run_id": TASK_RUN_ID2, "task_key": TASK_RUN_ID1_KEY, "start_time": 2}, + ] + trigger = DatabricksExecutionTrigger( + run_id=TASK_RUN_ID1, + databricks_conn_id=DEFAULT_CONN_ID, + polling_period_seconds=POLLING_INTERVAL_SECONDS, + run_page_url=RUN_PAGE_URL, + workflow_run_id=RUN_ID, + databricks_task_key=TASK_RUN_ID1_KEY, + ) + + await trigger.on_kill() + mock_get_run_tasks.assert_called_once_with(RUN_ID) + mock_cancel_run.assert_called_once_with(TASK_RUN_ID2) + class TestDatabricksSQLStatementExecutionTrigger: @pytest.fixture(autouse=True) diff --git a/providers/datadog/docs/changelog.rst b/providers/datadog/docs/changelog.rst index 9e80eb1f33083..15fd992c03879 100644 --- a/providers/datadog/docs/changelog.rst +++ b/providers/datadog/docs/changelog.rst @@ -203,7 +203,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.8.1 ..... diff --git a/providers/dbt/cloud/docs/changelog.rst b/providers/dbt/cloud/docs/changelog.rst index 6e24e6bb3581f..0b3276e48d791 100644 --- a/providers/dbt/cloud/docs/changelog.rst +++ b/providers/dbt/cloud/docs/changelog.rst @@ -433,7 +433,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.1.0 ..... diff --git a/providers/dingding/docs/changelog.rst b/providers/dingding/docs/changelog.rst index cd69acefb3f0e..5035430e909a9 100644 --- a/providers/dingding/docs/changelog.rst +++ b/providers/dingding/docs/changelog.rst @@ -206,7 +206,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.7.1 ..... diff --git a/providers/discord/docs/changelog.rst b/providers/discord/docs/changelog.rst index 3885f79c40cd4..59daa711a40dc 100644 --- a/providers/discord/docs/changelog.rst +++ b/providers/discord/docs/changelog.rst @@ -234,7 +234,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.9.1 ..... diff --git a/providers/docker/docs/changelog.rst b/providers/docker/docs/changelog.rst index 7d143964e3609..daf13c64354c3 100644 --- a/providers/docker/docs/changelog.rst +++ b/providers/docker/docs/changelog.rst @@ -331,7 +331,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.1.0 ..... diff --git a/providers/edge3/docs/changelog.rst b/providers/edge3/docs/changelog.rst index 75a1427dcaaad..abd3818ba3890 100644 --- a/providers/edge3/docs/changelog.rst +++ b/providers/edge3/docs/changelog.rst @@ -46,6 +46,7 @@ Misc * ``Bump the edge-ui-package-updates group across 1 directory with 3 updates (#70848)`` * ``Bump the edge-ui-package-updates group across 1 directory with 31 updates (#70616)`` * ``Add team_name tagging to multi team dag metrics (#69943)`` +* ``Bump @types/node to 26.1.2 (#70870)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): diff --git a/providers/edge3/src/airflow/providers/edge3/plugins/www/package.json b/providers/edge3/src/airflow/providers/edge3/plugins/www/package.json index ac5f6a386445a..eaa0310807630 100644 --- a/providers/edge3/src/airflow/providers/edge3/plugins/www/package.json +++ b/providers/edge3/src/airflow/providers/edge3/plugins/www/package.json @@ -56,7 +56,7 @@ "@testing-library/jest-dom": "^7.0.0", "@testing-library/react": "^16.3.2", "@trivago/prettier-plugin-sort-imports": "^6.0.2", - "@types/node": "^26.1.1", + "@types/node": "^26.1.2", "@types/react": "^19.2.17", "@types/react-dom": "^19.2.3", "@typescript-eslint/eslint-plugin": "8.65.0", diff --git a/providers/edge3/src/airflow/providers/edge3/plugins/www/pnpm-lock.yaml b/providers/edge3/src/airflow/providers/edge3/plugins/www/pnpm-lock.yaml index 8aa209e0f597d..efe14db4d961b 100644 --- a/providers/edge3/src/airflow/providers/edge3/plugins/www/pnpm-lock.yaml +++ b/providers/edge3/src/airflow/providers/edge3/plugins/www/pnpm-lock.yaml @@ -97,7 +97,7 @@ importers: version: 10.0.1(eslint@10.8.0(jiti@2.7.0)) '@rolldown/plugin-babel': specifier: ^0.2.3 - version: 0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@stylistic/eslint-plugin': specifier: ^5.10.0 version: 5.10.0(eslint@10.8.0(jiti@2.7.0)) @@ -111,8 +111,8 @@ importers: specifier: ^6.0.2 version: 6.0.2(prettier@3.9.6) '@types/node': - specifier: ^26.1.1 - version: 26.1.1 + specifier: ^26.1.2 + version: 26.1.2 '@types/react': specifier: ^19.2.17 version: 19.2.17 @@ -130,10 +130,10 @@ importers: version: 8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3) '@vitejs/plugin-react': specifier: ^6.0.4 - version: 6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitejs/plugin-react-swc': specifier: ^4.3.2 - version: 4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/coverage-v8': specifier: ^4.1.10 version: 4.1.10(vitest@4.1.10) @@ -184,16 +184,16 @@ importers: version: 8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3) vite: specifier: ^8.1.5 - version: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + version: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) vite-plugin-css-injected-by-js: specifier: ^5.0.2 - version: 5.0.2(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 5.0.2(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) vite-plugin-dts: specifier: ^5.0.3 - version: 5.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.1))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 5.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.2))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) vitest: specifier: ^4.1.10 - version: 4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + version: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) packages: @@ -630,36 +630,42 @@ packages: engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] + libc: [glibc] '@rolldown/binding-linux-arm64-musl@1.1.5': resolution: {integrity: sha512-lAryqH7IteztmCXQXk0etKj4wBQ7Gx5S6LjKhsgp9zb8I5bsuvU/2llH1hDQcjsFeqIsovMVN339/8pUDDBXxA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [arm64] os: [linux] + libc: [musl] '@rolldown/binding-linux-ppc64-gnu@1.1.5': resolution: {integrity: sha512-fsK/sNBnxzBlL4O1JNrZakVQxPspqpED5dLtNsZS9oOKmtSpdNIzxH2kkol5HYTWJN47sE20ztMJPxfZ89qGOg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [ppc64] os: [linux] + libc: [glibc] '@rolldown/binding-linux-s390x-gnu@1.1.5': resolution: {integrity: sha512-gLYb4BIadlfTOYT5gO503n8zQjXflgzpD0FcyKh0Mzx3rqCZKnHoJWV9xe1KXUJ5lx2JfcSHr/mhzS0PC/McAA==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [s390x] os: [linux] + libc: [glibc] '@rolldown/binding-linux-x64-gnu@1.1.5': resolution: {integrity: sha512-FjcpEKUyJygHgs1o50VYNvkt5+7Le/VEdYt0AkRpkL33MnyQfwr8l5mXwMmfmTbyMPr5vJLC+8/Gd9gXnwU1QQ==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] + libc: [glibc] '@rolldown/binding-linux-x64-musl@1.1.5': resolution: {integrity: sha512-Me+PfPI2TMeOQk0gYWfLQZtTktrmzbr8cDboqX83XKc7UrgAi55gF+2dUkWdxd19n55Essp2yeca+O9N5rBxHg==} engines: {node: ^20.19.0 || >=22.12.0} cpu: [x64] os: [linux] + libc: [musl] '@rolldown/binding-openharmony-arm64@1.1.5': resolution: {integrity: sha512-yc5WrLzXks6zCQfn9Oxr8pORKyl/pF+QjHmW/Qx3qu0oyrrNC+y2JLTU1E2rcWYAmzlnqngWXHQjy51VzW70Vw==} @@ -747,66 +753,79 @@ packages: resolution: {integrity: sha512-L+34Qqil+v5uC0zEubW7uByo78WOCIrBvci69E7sFASRl0X7b/MB6Cqd1lky/CtcSVTydWa2WZwFuWexjS5o6g==} cpu: [arm] os: [linux] + libc: [glibc] '@rollup/rollup-linux-arm-musleabihf@4.60.1': resolution: {integrity: sha512-n83O8rt4v34hgFzlkb1ycniJh7IR5RCIqt6mz1VRJD6pmhRi0CXdmfnLu9dIUS6buzh60IvACM842Ffb3xd6Gg==} cpu: [arm] os: [linux] + libc: [musl] '@rollup/rollup-linux-arm64-gnu@4.60.1': resolution: {integrity: sha512-Nql7sTeAzhTAja3QXeAI48+/+GjBJ+QmAH13snn0AJSNL50JsDqotyudHyMbO2RbJkskbMbFJfIJKWA6R1LCJQ==} cpu: [arm64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-arm64-musl@4.60.1': resolution: {integrity: sha512-+pUymDhd0ys9GcKZPPWlFiZ67sTWV5UU6zOJat02M1+PiuSGDziyRuI/pPue3hoUwm2uGfxdL+trT6Z9rxnlMA==} cpu: [arm64] os: [linux] + libc: [musl] '@rollup/rollup-linux-loong64-gnu@4.60.1': resolution: {integrity: sha512-VSvgvQeIcsEvY4bKDHEDWcpW4Yw7BtlKG1GUT4FzBUlEKQK0rWHYBqQt6Fm2taXS+1bXvJT6kICu5ZwqKCnvlQ==} cpu: [loong64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-loong64-musl@4.60.1': resolution: {integrity: sha512-4LqhUomJqwe641gsPp6xLfhqWMbQV04KtPp7/dIp0nzPxAkNY1AbwL5W0MQpcalLYk07vaW9Kp1PBhdpZYYcEw==} cpu: [loong64] os: [linux] + libc: [musl] '@rollup/rollup-linux-ppc64-gnu@4.60.1': resolution: {integrity: sha512-tLQQ9aPvkBxOc/EUT6j3pyeMD6Hb8QF2BTBnCQWP/uu1lhc9AIrIjKnLYMEroIz/JvtGYgI9dF3AxHZNaEH0rw==} cpu: [ppc64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-ppc64-musl@4.60.1': resolution: {integrity: sha512-RMxFhJwc9fSXP6PqmAz4cbv3kAyvD1etJFjTx4ONqFP9DkTkXsAMU4v3Vyc5BgzC+anz7nS/9tp4obsKfqkDHg==} cpu: [ppc64] os: [linux] + libc: [musl] '@rollup/rollup-linux-riscv64-gnu@4.60.1': resolution: {integrity: sha512-QKgFl+Yc1eEk6MmOBfRHYF6lTxiiiV3/z/BRrbSiW2I7AFTXoBFvdMEyglohPj//2mZS4hDOqeB0H1ACh3sBbg==} cpu: [riscv64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-riscv64-musl@4.60.1': resolution: {integrity: sha512-RAjXjP/8c6ZtzatZcA1RaQr6O1TRhzC+adn8YZDnChliZHviqIjmvFwHcxi4JKPSDAt6Uhf/7vqcBzQJy0PDJg==} cpu: [riscv64] os: [linux] + libc: [musl] '@rollup/rollup-linux-s390x-gnu@4.60.1': resolution: {integrity: sha512-wcuocpaOlaL1COBYiA89O6yfjlp3RwKDeTIA0hM7OpmhR1Bjo9j31G1uQVpDlTvwxGn2nQs65fBFL5UFd76FcQ==} cpu: [s390x] os: [linux] + libc: [glibc] '@rollup/rollup-linux-x64-gnu@4.60.1': resolution: {integrity: sha512-77PpsFQUCOiZR9+LQEFg9GClyfkNXj1MP6wRnzYs0EeWbPcHs02AXu4xuUbM1zhwn3wqaizle3AEYg5aeoohhg==} cpu: [x64] os: [linux] + libc: [glibc] '@rollup/rollup-linux-x64-musl@4.60.1': resolution: {integrity: sha512-5cIATbk5vynAjqqmyBjlciMJl1+R/CwX9oLk/EyiFXDWd95KpHdrOJT//rnUl4cUcskrd0jCCw3wpZnhIHdD9w==} cpu: [x64] os: [linux] + libc: [musl] '@rollup/rollup-openbsd-x64@4.60.1': resolution: {integrity: sha512-cl0w09WsCi17mcmWqqglez9Gk8isgeWvoUZ3WiJFYSR3zjBQc2J5/ihSjpl+VLjPqjQ/1hJRcqBfLjssREQILw==} @@ -900,36 +919,42 @@ packages: engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [glibc] '@swc/core-linux-arm64-musl@1.15.46': resolution: {integrity: sha512-imyRpNEcUzFQFV2LE4jL68ErvmKEuZCbvZru77iQREunJ+bR4i658cupTgtG1mLYM3F1Tzy3Sb9xYb02KghWTg==} engines: {node: '>=10'} cpu: [arm64] os: [linux] + libc: [musl] '@swc/core-linux-ppc64-gnu@1.15.46': resolution: {integrity: sha512-ctEfcl/HcUeomK33cbySiHZm98GEDIxTm1EkpBsYCiHxElYBzvTXVeuQT2YwbUXn9XCrjiw4ipyUNk33k26qRg==} engines: {node: '>=10'} cpu: [ppc64] os: [linux] + libc: [glibc] '@swc/core-linux-s390x-gnu@1.15.46': resolution: {integrity: sha512-DxlMdnt84TtRVTv7WL/thWyz9+QU8QZNNoAP9rrk0P68LziuhfePp8MjQ44zIprpTHTsEwyziIuGUUN5iSC1bQ==} engines: {node: '>=10'} cpu: [s390x] os: [linux] + libc: [glibc] '@swc/core-linux-x64-gnu@1.15.46': resolution: {integrity: sha512-SKxI7J6t90XPl8hRUqtJi9NfGdunN/E/vZMc7Bc0figeRdOPDBT+Tm8g7cx9xM0T0mewh2l+8dewa3Am27/P+A==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [glibc] '@swc/core-linux-x64-musl@1.15.46': resolution: {integrity: sha512-qj9T6B7bosI0VEsrWOVXZN1OXxS8Tp63ywyrLxNdOycnUtLdkgYcoBsN5y8ImnDDsnwrEWZOy1e+J4xSe7mA3Q==} engines: {node: '>=10'} cpu: [x64] os: [linux] + libc: [musl] '@swc/core-win32-arm64-msvc@1.15.46': resolution: {integrity: sha512-8p7l4c3LU+eA5g9Et1JPhNeMC1oQwXTGU+uah8DPIBX7YXzqswvaBtyKVmXefVGi/DJU1x3YJsc3mbAp9aWzSQ==} @@ -1055,8 +1080,8 @@ packages: '@types/json-schema@7.0.15': resolution: {integrity: sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==} - '@types/node@26.1.1': - resolution: {integrity: sha512-nxAkRSVkN1Y0JC1W8ky/fTfkGsMmcrRsbx+3XoZE+rMOX71kLYTV7fLXpqud1GpbpP5TuffXFqfX7fH2GgZREw==} + '@types/node@26.1.2': + resolution: {integrity: sha512-Vu4a5UFA9rIIFJ7rB/Vaafh9lrCQszopTCx6KjFboXTGQbPNasehVR5TEiithSDGyd1DEiUByggTZsg8jukeIg==} '@types/parse-json@4.0.2': resolution: {integrity: sha512-dISoDXWWQwUquiKsyZ4Ng+HX2KsPL7LyHKHQwgGFEA3IaKac4Obd+h2a/a6waisAoepJlBcx9paWqjA8/HVjCw==} @@ -2487,24 +2512,28 @@ packages: engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] + libc: [glibc] lightningcss-linux-arm64-musl@1.33.0: resolution: {integrity: sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==} engines: {node: '>= 12.0.0'} cpu: [arm64] os: [linux] + libc: [musl] lightningcss-linux-x64-gnu@1.33.0: resolution: {integrity: sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] + libc: [glibc] lightningcss-linux-x64-musl@1.33.0: resolution: {integrity: sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==} engines: {node: '>= 12.0.0'} cpu: [x64] os: [linux] + libc: [musl] lightningcss-win32-arm64-msvc@1.33.0: resolution: {integrity: sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==} @@ -3934,24 +3963,24 @@ snapshots: '@lukeed/ms@2.0.2': {} - '@microsoft/api-extractor-model@7.33.4(@types/node@26.1.1)': + '@microsoft/api-extractor-model@7.33.4(@types/node@26.1.2)': dependencies: '@microsoft/tsdoc': 0.16.0 '@microsoft/tsdoc-config': 0.18.1 - '@rushstack/node-core-library': 5.20.3(@types/node@26.1.1) + '@rushstack/node-core-library': 5.20.3(@types/node@26.1.2) transitivePeerDependencies: - '@types/node' optional: true - '@microsoft/api-extractor@7.57.6(@types/node@26.1.1)': + '@microsoft/api-extractor@7.57.6(@types/node@26.1.2)': dependencies: - '@microsoft/api-extractor-model': 7.33.4(@types/node@26.1.1) + '@microsoft/api-extractor-model': 7.33.4(@types/node@26.1.2) '@microsoft/tsdoc': 0.16.0 '@microsoft/tsdoc-config': 0.18.1 - '@rushstack/node-core-library': 5.20.3(@types/node@26.1.1) + '@rushstack/node-core-library': 5.20.3(@types/node@26.1.2) '@rushstack/rig-package': 0.7.2 - '@rushstack/terminal': 0.22.3(@types/node@26.1.1) - '@rushstack/ts-command-line': 5.3.3(@types/node@26.1.1) + '@rushstack/terminal': 0.22.3(@types/node@26.1.2) + '@rushstack/ts-command-line': 5.3.3(@types/node@26.1.2) diff: 9.0.0 lodash: 4.18.1 minimatch: 10.2.6 @@ -4048,14 +4077,14 @@ snapshots: '@rolldown/binding-win32-x64-msvc@1.1.5': optional: true - '@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0))': + '@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@babel/core': 8.0.1 picomatch: 4.0.4 rolldown: 1.1.5 optionalDependencies: '@babel/runtime': 7.29.7 - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) '@rolldown/pluginutils@1.0.1': {} @@ -4142,7 +4171,7 @@ snapshots: '@rollup/rollup-win32-x64-msvc@4.60.1': optional: true - '@rushstack/node-core-library@5.20.3(@types/node@26.1.1)': + '@rushstack/node-core-library@5.20.3(@types/node@26.1.2)': dependencies: ajv: 8.18.0 ajv-draft-04: 1.0.0(ajv@8.18.0) @@ -4153,12 +4182,12 @@ snapshots: resolve: 1.22.12 semver: 7.5.4 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 optional: true - '@rushstack/problem-matcher@0.2.1(@types/node@26.1.1)': + '@rushstack/problem-matcher@0.2.1(@types/node@26.1.2)': optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 optional: true '@rushstack/rig-package@0.7.2': @@ -4167,18 +4196,18 @@ snapshots: strip-json-comments: 3.1.1 optional: true - '@rushstack/terminal@0.22.3(@types/node@26.1.1)': + '@rushstack/terminal@0.22.3(@types/node@26.1.2)': dependencies: - '@rushstack/node-core-library': 5.20.3(@types/node@26.1.1) - '@rushstack/problem-matcher': 0.2.1(@types/node@26.1.1) + '@rushstack/node-core-library': 5.20.3(@types/node@26.1.2) + '@rushstack/problem-matcher': 0.2.1(@types/node@26.1.2) supports-color: 8.1.1 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 optional: true - '@rushstack/ts-command-line@5.3.3(@types/node@26.1.1)': + '@rushstack/ts-command-line@5.3.3(@types/node@26.1.2)': dependencies: - '@rushstack/terminal': 0.22.3(@types/node@26.1.1) + '@rushstack/terminal': 0.22.3(@types/node@26.1.2) '@types/argparse': 1.0.38 argparse: 1.0.10 string-argv: 0.3.2 @@ -4352,7 +4381,7 @@ snapshots: '@types/json-schema@7.0.15': {} - '@types/node@26.1.1': + '@types/node@26.1.2': dependencies: undici-types: 8.3.0 @@ -4372,7 +4401,7 @@ snapshots: '@types/ws@8.18.1': dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@typescript-eslint/eslint-plugin@8.65.0(@typescript-eslint/parser@8.65.0(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3))(eslint@10.8.0(jiti@2.7.0))(typescript@6.0.3)': dependencies: @@ -4467,20 +4496,20 @@ snapshots: '@typescript-eslint/types': 8.65.0 eslint-visitor-keys: 5.0.1 - '@vitejs/plugin-react-swc@4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0))': + '@vitejs/plugin-react-swc@4.3.2(@swc/helpers@0.5.23)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@rolldown/pluginutils': 1.0.1 '@swc/core': 1.15.46(@swc/helpers@0.5.23) - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) transitivePeerDependencies: - '@swc/helpers' - '@vitejs/plugin-react@6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0))': + '@vitejs/plugin-react@6.0.4(@rolldown/plugin-babel@0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)))(babel-plugin-react-compiler@1.0.0)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@rolldown/pluginutils': 1.0.1 - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) optionalDependencies: - '@rolldown/plugin-babel': 0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + '@rolldown/plugin-babel': 0.2.3(@babel/core@8.0.1)(@babel/runtime@7.29.7)(rolldown@1.1.5)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) babel-plugin-react-compiler: 1.0.0 '@vitest/coverage-v8@4.1.10(vitest@4.1.10)': @@ -4495,7 +4524,7 @@ snapshots: obug: 2.1.4 std-env: 4.2.0 tinyrainbow: 3.1.1 - vitest: 4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + vitest: 4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/expect@4.1.10': dependencies: @@ -4506,13 +4535,13 @@ snapshots: chai: 6.2.2 tinyrainbow: 3.1.1 - '@vitest/mocker@4.1.10(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0))': + '@vitest/mocker@4.1.10(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0))': dependencies: '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) '@vitest/pretty-format@4.1.10': dependencies: @@ -5308,7 +5337,7 @@ snapshots: buffer-image-size@0.6.4: dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 builtin-modules@5.3.0: {} @@ -5947,7 +5976,7 @@ snapshots: happy-dom@20.11.1: dependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@types/whatwg-mimetype': 3.0.2 '@types/ws': 8.18.1 buffer-image-size: 0.6.4 @@ -7048,7 +7077,7 @@ snapshots: universalify@2.0.1: optional: true - unplugin-dts@1.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.1))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)): + unplugin-dts@1.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.2))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: '@rollup/pluginutils': 5.4.0(rollup@4.60.1) '@volar/typescript': 2.4.28 @@ -7060,10 +7089,10 @@ snapshots: typescript: 6.0.3 unplugin: 2.3.11 optionalDependencies: - '@microsoft/api-extractor': 7.57.6(@types/node@26.1.1) + '@microsoft/api-extractor': 7.57.6(@types/node@26.1.2) rolldown: 1.1.5 rollup: 4.60.1 - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) transitivePeerDependencies: - supports-color @@ -7086,17 +7115,17 @@ snapshots: dependencies: react: 19.2.8 - vite-plugin-css-injected-by-js@5.0.2(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)): + vite-plugin-css-injected-by-js@5.0.2(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) - vite-plugin-dts@5.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.1))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)): + vite-plugin-dts@5.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.2))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: - unplugin-dts: 1.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.1))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + unplugin-dts: 1.0.3(@microsoft/api-extractor@7.57.6(@types/node@26.1.2))(rolldown@1.1.5)(rollup@4.60.1)(typescript@6.0.3)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) optionalDependencies: - '@microsoft/api-extractor': 7.57.6(@types/node@26.1.1) + '@microsoft/api-extractor': 7.57.6(@types/node@26.1.2) rollup: 4.60.1 - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) transitivePeerDependencies: - '@rspack/core' - '@vue/language-core' @@ -7106,7 +7135,7 @@ snapshots: - typescript - webpack - vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0): + vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0): dependencies: lightningcss: 1.33.0 picomatch: 4.0.5 @@ -7114,15 +7143,15 @@ snapshots: rolldown: 1.1.5 tinyglobby: 0.2.17 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 fsevents: 2.3.3 jiti: 2.7.0 yaml: 2.9.0 - vitest@4.1.10(@types/node@26.1.1)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)): + vitest@4.1.10(@types/node@26.1.2)(@vitest/coverage-v8@4.1.10)(happy-dom@20.11.1)(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)): dependencies: '@vitest/expect': 4.1.10 - '@vitest/mocker': 4.1.10(vite@8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0)) + '@vitest/mocker': 4.1.10(vite@8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0)) '@vitest/pretty-format': 4.1.10 '@vitest/runner': 4.1.10 '@vitest/snapshot': 4.1.10 @@ -7139,10 +7168,10 @@ snapshots: tinyexec: 1.2.4 tinyglobby: 0.2.17 tinyrainbow: 3.1.1 - vite: 8.1.5(@types/node@26.1.1)(jiti@2.7.0)(yaml@2.9.0) + vite: 8.1.5(@types/node@26.1.2)(jiti@2.7.0)(yaml@2.9.0) why-is-node-running: 2.3.0 optionalDependencies: - '@types/node': 26.1.1 + '@types/node': 26.1.2 '@vitest/coverage-v8': 4.1.10(vitest@4.1.10) happy-dom: 20.11.1 transitivePeerDependencies: diff --git a/providers/elasticsearch/docs/changelog.rst b/providers/elasticsearch/docs/changelog.rst index f009e0f5b57e1..6d7b9a2af4b4b 100644 --- a/providers/elasticsearch/docs/changelog.rst +++ b/providers/elasticsearch/docs/changelog.rst @@ -467,7 +467,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 6.1.0 diff --git a/providers/exasol/README.rst b/providers/exasol/README.rst index 76470f4748039..70aba90425d70 100644 --- a/providers/exasol/README.rst +++ b/providers/exasol/README.rst @@ -50,17 +50,17 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``pyexasol`` ``>=0.26.0,<2`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= Optional dependencies ---------------------- diff --git a/providers/exasol/docs/changelog.rst b/providers/exasol/docs/changelog.rst index 2738dff387b03..3a22306fd40c8 100644 --- a/providers/exasol/docs/changelog.rst +++ b/providers/exasol/docs/changelog.rst @@ -33,10 +33,11 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 4.10.4 ...... @@ -303,7 +304,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.7.1 ..... diff --git a/providers/exasol/docs/index.rst b/providers/exasol/docs/index.rst index e15fc678b293a..d361cc38f9db5 100644 --- a/providers/exasol/docs/index.rst +++ b/providers/exasol/docs/index.rst @@ -95,17 +95,17 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``pyexasol`` ``>=0.26.0,<2`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= Optional dependencies --------------------- diff --git a/providers/exasol/pyproject.toml b/providers/exasol/pyproject.toml index 06bee594aa659..a994d8c4d43c5 100644 --- a/providers/exasol/pyproject.toml +++ b/providers/exasol/pyproject.toml @@ -65,18 +65,9 @@ dependencies = [ # Capped to 1.x: pyexasol 2.x ships stricter types that break the exasol hook. # Remove the cap after migrating; tracked at https://github.com/apache/airflow/issues/69123 "pyexasol>=0.26.0,<2", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] # The optional dependencies should be modified in place in the generated file diff --git a/providers/fab/docs/changelog.rst b/providers/fab/docs/changelog.rst index 0efce33768491..8ec622d56d2e3 100644 --- a/providers/fab/docs/changelog.rst +++ b/providers/fab/docs/changelog.rst @@ -34,6 +34,7 @@ Bug Fixes ~~~~~~~~~ * ``Fix asset state store writes forbidden under FAB auth manager (#70478)`` +* ``Stop dumping every Dag to stdout when syncing FAB permissions (#70939)`` Misc ~~~~ diff --git a/providers/fab/src/airflow/providers/fab/www/package.json b/providers/fab/src/airflow/providers/fab/www/package.json index 0598d8be27b22..15a7c18c6b4be 100644 --- a/providers/fab/src/airflow/providers/fab/www/package.json +++ b/providers/fab/src/airflow/providers/fab/www/package.json @@ -56,8 +56,8 @@ "stylelint": "^17.14.1", "terser-webpack-plugin": "^5.6.1", "url-loader": "4.1.1", - "webpack": "^5.109.0", - "webpack-cli": "^7.2.1", + "webpack": "^5.109.2", + "webpack-cli": "^7.2.2", "webpack-license-plugin": "^4.5.1", "webpack-manifest-plugin": "^6.0.1" }, diff --git a/providers/fab/src/airflow/providers/fab/www/pnpm-lock.yaml b/providers/fab/src/airflow/providers/fab/www/pnpm-lock.yaml index 7cd058e0277e3..2e03dc53b3707 100644 --- a/providers/fab/src/airflow/providers/fab/www/pnpm-lock.yaml +++ b/providers/fab/src/airflow/providers/fab/www/pnpm-lock.yaml @@ -42,31 +42,31 @@ importers: version: 8.0.2(@babel/core@8.0.1) babel-loader: specifier: ^10.1.1 - version: 10.1.1(@babel/core@8.0.1)(webpack@5.109.0) + version: 10.1.1(@babel/core@8.0.1)(webpack@5.109.2) copy-webpack-plugin: specifier: ^14.0.0 - version: 14.0.0(webpack@5.109.0) + version: 14.0.0(webpack@5.109.2) css-loader: specifier: 7.1.4 - version: 7.1.4(webpack@5.109.0) + version: 7.1.4(webpack@5.109.2) css-minimizer-webpack-plugin: specifier: ^8.0.0 - version: 8.0.0(webpack@5.109.0) + version: 8.0.0(webpack@5.109.2) eslint: specifier: ^10.8.0 version: 10.8.0 file-loader: specifier: ^6.2.0 - version: 6.2.0(webpack@5.109.0) + version: 6.2.0(webpack@5.109.2) mini-css-extract-plugin: specifier: ^2.10.2 - version: 2.10.2(webpack@5.109.0) + version: 2.10.2(webpack@5.109.2) moment: specifier: ^2.30.1 version: 2.30.1 moment-locales-webpack-plugin: specifier: ^1.2.0 - version: 1.2.0(moment@2.30.1)(webpack@5.109.0) + version: 1.2.0(moment@2.30.1)(webpack@5.109.2) prettier: specifier: ^3.9.6 version: 3.9.6 @@ -75,22 +75,22 @@ importers: version: 17.14.1 terser-webpack-plugin: specifier: ^5.6.1 - version: 5.6.1(postcss@8.5.24)(webpack@5.109.0) + version: 5.6.1(postcss@8.5.24)(webpack@5.109.2) url-loader: specifier: 4.1.1 - version: 4.1.1(file-loader@6.2.0(webpack@5.109.0))(webpack@5.109.0) + version: 4.1.1(file-loader@6.2.0(webpack@5.109.2))(webpack@5.109.2) webpack: - specifier: ^5.109.0 - version: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + specifier: ^5.109.2 + version: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) webpack-cli: - specifier: ^7.2.1 - version: 7.2.1(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.0) + specifier: ^7.2.2 + version: 7.2.2(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.2) webpack-license-plugin: specifier: ^4.5.1 - version: 4.5.1(webpack@5.109.0) + version: 4.5.1(webpack@5.109.2) webpack-manifest-plugin: specifier: ^6.0.1 - version: 6.0.1(webpack@5.109.0) + version: 6.0.1(webpack@5.109.2) packages: @@ -918,8 +918,8 @@ packages: resolution: {integrity: sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==} engines: {node: 18 || 20 || >=22} - baseline-browser-mapping@2.11.5: - resolution: {integrity: sha512-xJo6a6YZnwZfnyGmQKWMbVOcii7XRibjOskRh+WJ9UHQoX16xrQrcIgAMQOzfvs8XiLMx6ih/fsLPF73iY2D1A==} + baseline-browser-mapping@2.11.12: + resolution: {integrity: sha512-r7WnVImvVCeFpf2DOXfy41aPWzeNg3H/A2X4dKmy1QL0MSyyk/e7z8ihJ3N6Nn2PsdhkVlqnEfnUE4a05P2aTA==} engines: {node: '>=6.0.0'} hasBin: true @@ -955,8 +955,8 @@ packages: caniuse-api@3.0.0: resolution: {integrity: sha512-bsTwuIg/BZZK/vreVTYYbSWoe2F+71P7K5QGEX+pT250DZbfU1MQ5prOKpPR+LL6uWKK3KMwMCAS74QB3Um1uw==} - caniuse-lite@1.0.30001806: - resolution: {integrity: sha512-72Cuvd95zbSYPKq6Fhg8eDJRlzgWDf7/mtoZv6Qe/DYNCEBdNxoA3+rZAU2ZhGCpZlns3EssFavaZomckT5Uuw==} + caniuse-lite@1.0.30001809: + resolution: {integrity: sha512-xxWVywk6a6Arlk+hymeycyn/VgqEfLDxupvhH/xiY5SJ/18kmi9o6MiO320DCUzypORHLtvh0I4i04tUhCNHNQ==} chalk@4.1.2: resolution: {integrity: sha512-oKnbhFyRIXpUuez8iBMmyEa4nbj4IOQyuhc/wy9kY7/WVPcwIO9VA668Pu8RkO7+0G76SLROeyw9CpQ061i4mA==} @@ -1138,8 +1138,8 @@ packages: domutils@3.2.2: resolution: {integrity: sha512-6kZKyUajlDuqlHKVX1w7gyslj9MPIXzIFiz/rGu35uC1wMi+kMhQwGhl4lt9unC9Vb9INnY9Z3/ZA3+FhASLaw==} - electron-to-chromium@1.5.397: - resolution: {integrity: sha512-khGTy9U9x02KEtsKM8vx5A62BsRmcOsIgDpWr1ImE32Ax8GxHGPHZf+Eu9H8zOOyHJnB0jTbseyTHbq2XCT8yw==} + electron-to-chromium@1.5.402: + resolution: {integrity: sha512-/oOpMaPT6Yg+6/1XQhyIPlzgj7Ye9zf+nNM2Uh6OcE2G2oNptWazFa+qB2Pdqqbsc9KnIDzgAntoYN0dbwOXwA==} emoji-regex@8.0.0: resolution: {integrity: sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==} @@ -1152,8 +1152,8 @@ packages: resolution: {integrity: sha512-YGRs8knHhKHVShLkFET/rWAU8kmHbOV5LwN938RHI0pljAJ1Gf6SzXsSmRaEzcXTtOOmVqJ5+WtQPL5uigY50Q==} engines: {node: '>=14'} - enhanced-resolve@5.24.3: - resolution: {integrity: sha512-PwKooW9JUzh5chmYfHM3IQl5OkK2u2Nm011MgeZrss3JmFraUx/fqrf78kk8GUMYoibx/14MdwTl/1WKkG7TpQ==} + enhanced-resolve@5.24.5: + resolution: {integrity: sha512-L1l8TNvomm6UVW5B253AGxQagSQr+vGwhMlrrfRS2qmhx46AMpMVJKQYLvWYbysTMY8VoicOvzHzoHMbyzB+4A==} engines: {node: '>=10.13.0'} entities@4.5.0: @@ -1254,8 +1254,8 @@ packages: fast-levenshtein@2.0.6: resolution: {integrity: sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==} - fast-uri@3.1.4: - resolution: {integrity: sha512-8JnbkQ4juDyvYs4mgFGQqg4yCYtFDtUtmp2QIQq11ZZe5CFQ5wcqm1rqDgAh/QdMySuBnPzMUiJUNZG5N/AiQw==} + fast-uri@3.1.5: + resolution: {integrity: sha512-gHwA1O9LDIcKunMKhObS/HimwtehO1nPUECKAu5TpKgaO19fcWEl4bliWe1jWxVFvIXztJjjQ4L8XQ1EU9f7Jw==} fastest-levenshtein@1.0.16: resolution: {integrity: sha512-eRnCtTTtGZFpQCwhJiUOuxPQWRXVKYDn0b2PeHfXL6/Zi53SLAzAHfVhVWK2AryC/WH05kGfxhFIPvTF0SXQzg==} @@ -1691,8 +1691,8 @@ packages: neo-async@2.6.2: resolution: {integrity: sha512-Yd3UES5mWCSqR+qNT93S3UoYUkqAZ9lLg8a7g9rimsWmYGK8cVToA4/sF3RrshdyV3sAGMXVUmpMYOw+dLpOuw==} - node-releases@2.0.51: - resolution: {integrity: sha512-wRNIrw4DmVLKQlbgOMdkMx27Wrpzes2hh5Jtbi2bjPd+4wJstWIqP5A+lscnqbm0xxmT5Bpg8Lec5ItEBwx6BQ==} + node-releases@2.0.53: + resolution: {integrity: sha512-D9UOmYG3UH1V+ENW56t5QXBwJw1YEY18ruVeus89Rw+SyIgjPkCO84bRzO3uNIYosJbNwiabWVn48o3uJLjxFQ==} engines: {node: '>=18'} normalize-path@3.0.0: @@ -2240,8 +2240,8 @@ packages: engines: {node: '>=10'} hasBin: true - terser@5.49.0: - resolution: {integrity: sha512-SNiDnXyHSrxVcIOtVbULzcTmniUiwcV7Nwdyj1twVubeTmbjoa8p69KKDpfkdoOavuM4/GRm1+ykI8qqnavHoA==} + terser@5.49.2: + resolution: {integrity: sha512-rGbJiKeQ4WDe3EXlDAIaQcwftVfv2Q8o1awFNfvXolJYKkb1AuZY1RTOmqx4LJXZENbWZA7eIsYGHuEzHsi1nQ==} engines: {node: '>=10'} hasBin: true @@ -2283,8 +2283,8 @@ packages: resolution: {integrity: sha512-wH590V9VNgYH9g3lH9wWjTrUoKsjLF6sGLjhR4sH1LWpLmCOH0Zf7PukhDA8BiS7KHe4oPNkcTHqYkj7SOGUOw==} engines: {node: '>=20'} - update-browserslist-db@1.2.3: - resolution: {integrity: sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==} + update-browserslist-db@1.3.0: + resolution: {integrity: sha512-x/M6q3w4Ybp91CNaS4S69UnliqR3BzRpOT6LWbksjth0S/+jhfaPJsWjt/TewpT8j9eLIojUf5jr29WextHroA==} hasBin: true peerDependencies: browserslist: '>= 4.21.0' @@ -2309,8 +2309,8 @@ packages: resolution: {integrity: sha512-6i/00NBjP4yGPs+caKSyRfpTF/8Torsu0MOW3mMzIbhgISFder8i7xbqgHlLMwJrdiN8ndBV3UA1/AfzPSr+jg==} engines: {node: '>=10.13.0'} - webpack-cli@7.2.1: - resolution: {integrity: sha512-YwSGbcZdfz12DM8JIseVPr3oBb09IgVCVc4vY3oDvZnI/mALTGPAP1QiqOi4/bBLSJrRHaqDIXeHcNA0+G38aw==} + webpack-cli@7.2.2: + resolution: {integrity: sha512-lD0pALneslq8FfV+rwvm1BMW0AFAJrHHhNupAGN4asYjMvqrtRsenU4iKpiBo09gS4ntMxKGUxl9jhTEzVt0oA==} engines: {node: '>=20.9.0'} hasBin: true peerDependencies: @@ -2356,8 +2356,8 @@ packages: resolution: {integrity: sha512-jyuiGJdtvY434z5bUZrjz67v76/ePNvFZTp9Mdz29IlH4+GPsgyGjiv0fKI+M7BdkU6ADjulUcKAd3tUK3WlEw==} engines: {node: '>=10.13.0'} - webpack@5.109.0: - resolution: {integrity: sha512-vomrngskVVXEZF9sMZfYAd4pXZUnfaWdJGlF+BTNF+gJBCKYCQBnOeVPlrh39Ewl7nlCsirDplMy6o5g9xJHBg==} + webpack@5.109.2: + resolution: {integrity: sha512-U9/cvLzxObKNEZ9+TtdqrHM5/9z3lgl2c+c4BzbqGxFQvQvBAq87yql5A8pQ+rrMbS496MZJeF5enVBndIy2hw==} engines: {node: '>=10.13.0'} hasBin: true peerDependencies: @@ -3270,7 +3270,7 @@ snapshots: ajv@8.20.0: dependencies: fast-deep-equal: 3.1.3 - fast-uri: 3.1.4 + fast-uri: 3.1.5 json-schema-traverse: 1.0.0 require-from-string: 2.0.2 @@ -3286,12 +3286,12 @@ snapshots: astral-regex@2.0.0: {} - babel-loader@10.1.1(@babel/core@8.0.1)(webpack@5.109.0): + babel-loader@10.1.1(@babel/core@8.0.1)(webpack@5.109.2): dependencies: '@babel/core': 8.0.1 find-up: 5.0.0 optionalDependencies: - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) babel-plugin-polyfill-corejs3@1.0.0(@babel/core@8.0.1): dependencies: @@ -3301,7 +3301,7 @@ snapshots: balanced-match@4.0.4: {} - baseline-browser-mapping@2.11.5: {} + baseline-browser-mapping@2.11.12: {} big.js@5.2.2: {} @@ -3317,11 +3317,11 @@ snapshots: browserslist@4.28.7: dependencies: - baseline-browser-mapping: 2.11.5 - caniuse-lite: 1.0.30001806 - electron-to-chromium: 1.5.397 - node-releases: 2.0.51 - update-browserslist-db: 1.2.3(browserslist@4.28.7) + baseline-browser-mapping: 2.11.12 + caniuse-lite: 1.0.30001809 + electron-to-chromium: 1.5.402 + node-releases: 2.0.53 + update-browserslist-db: 1.3.0(browserslist@4.28.7) buffer-from@1.1.2: {} @@ -3338,11 +3338,11 @@ snapshots: caniuse-api@3.0.0: dependencies: browserslist: 4.28.7 - caniuse-lite: 1.0.30001806 + caniuse-lite: 1.0.30001809 lodash.memoize: 4.1.2 lodash.uniq: 4.5.0 - caniuse-lite@1.0.30001806: {} + caniuse-lite@1.0.30001809: {} chalk@4.1.2: dependencies: @@ -3377,14 +3377,14 @@ snapshots: convert-source-map@2.0.0: {} - copy-webpack-plugin@14.0.0(webpack@5.109.0): + copy-webpack-plugin@14.0.0(webpack@5.109.2): dependencies: glob-parent: 6.0.2 normalize-path: 3.0.0 schema-utils: 4.3.3 serialize-javascript: 7.0.5 tinyglobby: 0.2.15 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) core-js-compat@3.49.0: dependencies: @@ -3409,7 +3409,7 @@ snapshots: css-functions-list@3.3.3: {} - css-loader@7.1.4(webpack@5.109.0): + css-loader@7.1.4(webpack@5.109.2): dependencies: icss-utils: 5.1.0(postcss@8.5.12) postcss: 8.5.12 @@ -3420,9 +3420,9 @@ snapshots: postcss-value-parser: 4.2.0 semver: 7.7.4 optionalDependencies: - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) - css-minimizer-webpack-plugin@8.0.0(webpack@5.109.0): + css-minimizer-webpack-plugin@8.0.0(webpack@5.109.2): dependencies: '@jridgewell/trace-mapping': 0.3.31 cssnano: 7.1.3(postcss@8.5.12) @@ -3430,7 +3430,7 @@ snapshots: postcss: 8.5.12 schema-utils: 4.3.3 serialize-javascript: 7.0.5 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) css-select@5.2.2: dependencies: @@ -3526,7 +3526,7 @@ snapshots: domelementtype: 2.3.0 domhandler: 5.0.3 - electron-to-chromium@1.5.397: {} + electron-to-chromium@1.5.402: {} emoji-regex@8.0.0: {} @@ -3534,7 +3534,7 @@ snapshots: empathic@2.0.1: {} - enhanced-resolve@5.24.3: + enhanced-resolve@5.24.5: dependencies: graceful-fs: 4.2.11 tapable: 2.3.3 @@ -3644,7 +3644,7 @@ snapshots: fast-levenshtein@2.0.6: {} - fast-uri@3.1.4: {} + fast-uri@3.1.5: {} fastest-levenshtein@1.0.16: {} @@ -3664,11 +3664,11 @@ snapshots: dependencies: flat-cache: 4.0.1 - file-loader@6.2.0(webpack@5.109.0): + file-loader@6.2.0(webpack@5.109.2): dependencies: loader-utils: 2.0.4 schema-utils: 3.3.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) fill-range@7.1.1: dependencies: @@ -3935,31 +3935,31 @@ snapshots: dependencies: mime-db: 1.52.0 - mini-css-extract-plugin@2.10.2(webpack@5.109.0): + mini-css-extract-plugin@2.10.2(webpack@5.109.2): dependencies: schema-utils: 4.3.3 tapable: 2.3.2 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) minimatch@10.2.6: dependencies: brace-expansion: 5.0.8 - minimizer-webpack-plugin@5.6.1(postcss@8.5.24)(webpack@5.109.0): + minimizer-webpack-plugin@5.6.1(postcss@8.5.24)(webpack@5.109.2): dependencies: '@jridgewell/trace-mapping': 0.3.31 jest-worker: 27.5.1 schema-utils: 4.3.3 - terser: 5.49.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + terser: 5.49.2 + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) optionalDependencies: postcss: 8.5.24 - moment-locales-webpack-plugin@1.2.0(moment@2.30.1)(webpack@5.109.0): + moment-locales-webpack-plugin@1.2.0(moment@2.30.1)(webpack@5.109.2): dependencies: lodash.difference: 4.5.0 moment: 2.30.1 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) moment-timezone@0.6.3: dependencies: @@ -3982,7 +3982,7 @@ snapshots: neo-async@2.6.2: {} - node-releases@2.0.51: {} + node-releases@2.0.53: {} normalize-path@3.0.0: {} @@ -4475,13 +4475,13 @@ snapshots: tapable@2.3.3: {} - terser-webpack-plugin@5.6.1(postcss@8.5.24)(webpack@5.109.0): + terser-webpack-plugin@5.6.1(postcss@8.5.24)(webpack@5.109.2): dependencies: '@jridgewell/trace-mapping': 0.3.31 jest-worker: 27.5.1 schema-utils: 4.3.3 terser: 5.48.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) optionalDependencies: postcss: 8.5.24 @@ -4492,7 +4492,7 @@ snapshots: commander: 2.20.3 source-map-support: 0.5.21 - terser@5.49.0: + terser@5.49.2: dependencies: '@jridgewell/source-map': 0.3.11 acorn: 8.18.0 @@ -4529,7 +4529,7 @@ snapshots: unicorn-magic@0.4.0: {} - update-browserslist-db@1.2.3(browserslist@4.28.7): + update-browserslist-db@1.3.0(browserslist@4.28.7): dependencies: browserslist: 4.28.7 escalade: 3.2.0 @@ -4539,14 +4539,14 @@ snapshots: dependencies: punycode: 2.3.1 - url-loader@4.1.1(file-loader@6.2.0(webpack@5.109.0))(webpack@5.109.0): + url-loader@4.1.1(file-loader@6.2.0(webpack@5.109.2))(webpack@5.109.2): dependencies: loader-utils: 2.0.4 mime-types: 2.1.35 schema-utils: 3.3.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) optionalDependencies: - file-loader: 6.2.0(webpack@5.109.0) + file-loader: 6.2.0(webpack@5.109.2) util-deprecate@1.0.2: {} @@ -4554,7 +4554,7 @@ snapshots: dependencies: graceful-fs: 4.2.11 - webpack-cli@7.2.1(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.0): + webpack-cli@7.2.2(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.2): dependencies: '@discoveryjs/json-ext': 1.1.0 commander: 14.0.3 @@ -4563,25 +4563,25 @@ snapshots: import-local: 3.2.0 interpret: 3.1.1 rechoir: 0.8.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) webpack-merge: 6.0.1 optionalDependencies: js-yaml: 4.3.0 json5: 2.2.3 - webpack-license-plugin@4.5.1(webpack@5.109.0): + webpack-license-plugin@4.5.1(webpack@5.109.2): dependencies: chalk: 5.6.2 lodash: 4.18.1 needle: 3.3.1 spdx-expression-validate: 2.0.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) webpack-sources: 3.3.4 - webpack-manifest-plugin@6.0.1(webpack@5.109.0): + webpack-manifest-plugin@6.0.1(webpack@5.109.2): dependencies: tapable: 2.3.0 - webpack: 5.109.0(postcss@8.5.24)(webpack-cli@7.2.1) + webpack: 5.109.2(postcss@8.5.24)(webpack-cli@7.2.2) webpack-sources: 3.3.4 webpack-merge@6.0.1: @@ -4594,7 +4594,7 @@ snapshots: webpack-sources@3.5.1: {} - webpack@5.109.0(postcss@8.5.24)(webpack-cli@7.2.1): + webpack@5.109.2(postcss@8.5.24)(webpack-cli@7.2.2): dependencies: '@types/estree': 1.0.9 '@types/json-schema': 7.0.15 @@ -4604,20 +4604,20 @@ snapshots: acorn: 8.18.0 browserslist: 4.28.7 chrome-trace-event: 1.0.4 - enhanced-resolve: 5.24.3 + enhanced-resolve: 5.24.5 es-module-lexer: 2.3.1 eslint-scope: 5.1.1 events: 3.3.0 graceful-fs: 4.2.11 mime-db: 1.54.0 - minimizer-webpack-plugin: 5.6.1(postcss@8.5.24)(webpack@5.109.0) + minimizer-webpack-plugin: 5.6.1(postcss@8.5.24)(webpack@5.109.2) neo-async: 2.6.2 schema-utils: 4.3.3 tapable: 2.3.3 watchpack: 2.5.2 webpack-sources: 3.5.1 optionalDependencies: - webpack-cli: 7.2.1(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.0) + webpack-cli: 7.2.2(js-yaml@4.3.0)(json5@2.2.3)(webpack@5.109.2) transitivePeerDependencies: - '@minify-html/node' - '@swc/core' diff --git a/providers/ftp/docs/changelog.rst b/providers/ftp/docs/changelog.rst index bed31311e5f7c..6857793206504 100644 --- a/providers/ftp/docs/changelog.rst +++ b/providers/ftp/docs/changelog.rst @@ -226,7 +226,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.12.1 ...... diff --git a/providers/git/docs/changelog.rst b/providers/git/docs/changelog.rst index ec2c000d526d9..2291b0e5a0bfd 100644 --- a/providers/git/docs/changelog.rst +++ b/providers/git/docs/changelog.rst @@ -27,6 +27,11 @@ Bug Fixes * ``Use private key when pulling submodules (#62938)`` +Misc +~~~~ + +* ``Use ValueError for conflicting Git connection key arguments (#70538)`` + .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): diff --git a/providers/git/src/airflow/providers/git/bundles/git.py b/providers/git/src/airflow/providers/git/bundles/git.py index f7b8de1a4a4c5..917e3e6a3335d 100644 --- a/providers/git/src/airflow/providers/git/bundles/git.py +++ b/providers/git/src/airflow/providers/git/bundles/git.py @@ -271,7 +271,7 @@ def _clone_repo_if_required(self) -> None: self.repo = Repo(self.repo_path) except NoSuchPathError as e: # Protection should the bare repo be removed manually - raise AirflowException("Repository path: %s not found", self.bare_repo_path) from e + raise FileNotFoundError(f"Repository path: {self.bare_repo_path} not found") from e except (InvalidGitRepositoryError, GitCommandError) as e: self._log.warning( "Repository clone/open failed, cleaning up and retrying", diff --git a/providers/git/src/airflow/providers/git/hooks/git.py b/providers/git/src/airflow/providers/git/hooks/git.py index 4ec8738758b7c..0eafece941a8f 100644 --- a/providers/git/src/airflow/providers/git/hooks/git.py +++ b/providers/git/src/airflow/providers/git/hooks/git.py @@ -30,7 +30,7 @@ from urllib.parse import quote as urlquote from airflow.exceptions import AirflowProviderDeprecationWarning -from airflow.providers.common.compat.sdk import AirflowException, BaseHook +from airflow.providers.common.compat.sdk import BaseHook log = logging.getLogger(__name__) @@ -113,7 +113,7 @@ def __init__( self.env: dict[str, str] = {} if self.key_file and self.private_key: - raise AirflowException("Both 'key_file' and 'private_key' cannot be provided at the same time") + raise ValueError("Both 'key_file' and 'private_key' cannot be provided at the same time") if host_key_checking_defaulted and self._uses_ssh_transport_options(): warnings.warn( diff --git a/providers/git/tests/unit/git/bundles/test_git.py b/providers/git/tests/unit/git/bundles/test_git.py index 6826ae76b6c9b..0726d0a5fc8f4 100644 --- a/providers/git/tests/unit/git/bundles/test_git.py +++ b/providers/git/tests/unit/git/bundles/test_git.py @@ -1234,10 +1234,10 @@ def test_clone_repo_no_such_path_error(self, mock_githook): with mock.patch("airflow.providers.git.bundles.git.Repo.clone_from") as mock_clone: mock_clone.side_effect = NoSuchPathError("Path not found") bundle = GitDagBundle(name="test", tracking_ref="main") - with pytest.raises(AirflowException) as exc_info: + with pytest.raises(FileNotFoundError) as exc_info: bundle._clone_repo_if_required() - assert "Repository path: %s not found" in str(exc_info.value) + assert str(exc_info.value) == f"Repository path: {bundle.bare_repo_path} not found" @patch.dict(os.environ, {"AIRFLOW_CONN_MY_TEST_GIT": '{"host": "something", "conn_type": "git"}'}) @pytest.mark.parametrize( diff --git a/providers/git/tests/unit/git/hooks/test_git.py b/providers/git/tests/unit/git/hooks/test_git.py index 1d750ca99c37a..484791fe19c34 100644 --- a/providers/git/tests/unit/git/hooks/test_git.py +++ b/providers/git/tests/unit/git/hooks/test_git.py @@ -26,7 +26,6 @@ from airflow.exceptions import AirflowProviderDeprecationWarning from airflow.models import Connection -from airflow.providers.common.compat.sdk import AirflowException from airflow.providers.git.hooks.git import GitHook from tests_common.test_utils.config import conf_vars @@ -204,7 +203,7 @@ def test_given_both_private_key_and_key_file(self, create_connection_without_db) ) with pytest.raises( - AirflowException, match="Both 'key_file' and 'private_key' cannot be provided at the same time" + ValueError, match="Both 'key_file' and 'private_key' cannot be provided at the same time" ): GitHook(git_conn_id=CONN_BOTH_PATH_INLINE) diff --git a/providers/github/docs/changelog.rst b/providers/github/docs/changelog.rst index 80f81505ab522..ab3c0b250c43a 100644 --- a/providers/github/docs/changelog.rst +++ b/providers/github/docs/changelog.rst @@ -238,7 +238,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 2.8.1 ..... diff --git a/providers/google/README.rst b/providers/google/README.rst index bd1c048e1a7e1..bb7bdc4eca0b8 100644 --- a/providers/google/README.rst +++ b/providers/google/README.rst @@ -57,9 +57,9 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.13.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` @@ -77,7 +77,14 @@ PIP package Version required ``google-auth`` ``>=2.29.0`` ``google-auth-httplib2`` ``>=0.0.1`` ``google-genai`` ``>=2.8.0`` -``google-cloud-aiplatform[evaluation]`` ``>=1.155.0`` +``google-cloud-aiplatform`` ``>=1.155.0`` +``tqdm`` ``>=4.23.0`` +``scikit-learn`` ``>=1.6.0`` +``jsonschema`` ``>=4.19.1`` +``ruamel.yaml`` ``>=0.19.0`` +``pyyaml`` ``>=6.0.3`` +``litellm`` ``>=1.83.7,<1.86.0; python_version < "3.14"`` +``litellm`` ``>=1.93.0; python_version >= "3.14"`` ``ray[default]`` ``>=2.42.0; python_version < "3.13"`` ``ray[default]`` ``>=2.49.0; python_version >= "3.13" and python_version < "3.14"`` ``ray[default]`` ``>=2.55.0; python_version >= "3.14" and python_version < "3.15"`` @@ -125,9 +132,9 @@ PIP package Version required ``httpx`` ``>=0.25.0`` ``looker-sdk`` ``>=22.4.0,!=24.18.0`` ``pandas-gbq`` ``>=0.7.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``proto-plus`` ``>=1.26.0`` ``pyarrow`` ``>=18.0.0; python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` @@ -138,7 +145,7 @@ PIP package Version required ``tenacity`` ``>=8.3.0`` ``immutabledict`` ``>=4.2.0`` ``types-protobuf`` ``>=5.27.0,!=5.29.1.20250402`` -========================================== ==================================================================== +========================================== ================================================================== Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/google/docs/changelog.rst b/providers/google/docs/changelog.rst index 8d3603296bd97..701265bc1e949 100644 --- a/providers/google/docs/changelog.rst +++ b/providers/google/docs/changelog.rst @@ -68,6 +68,7 @@ Bug Fixes * ``Emit GCSListObjectsOperator delimiter deprecation warning after rendering (#70533)`` * ``Move Compute Engine validation out of constructors (#70454)`` * ``Prevent premature validation in Gemini batch job operators (#70362)`` +* ``Only refuse team scoped like secret ids when multi_team is on (#71078)`` Misc ~~~~ @@ -75,16 +76,17 @@ Misc * ``Make psycopg (v3) the default synchronous Postgres driver (#69526)`` * ``Flag conn-fields in hook but absent from provider.yaml in static checks (#69655)`` * ``Strip only the literal properties/ prefix from GA property link IDs (#70919)`` -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Update system test to the latest api version of google-ads (#69865)`` * ``Use common.compat.sdk for timezone imports in providers (#70492)`` * ``Drop redundant BigQueryInsertJobOperator job ID snapshot in __init__ (#70494)`` +* ``Install google-cloud-aiplatform without the 'evaluation' extra and require its dependencies directly, so the provider installs on Python 3.14 (#71273)`` Doc-only ~~~~~~~~ * ``Link task state store docs in durable execution across providers (#69851)`` * ``Document effect of state-store cleanup for operators with durable execution (#70721)`` +* ``Fix some docs and test gaps following up multi team secret refusal (#71106)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): @@ -93,6 +95,11 @@ Doc-only * ``Preserve Dataform workflow invocation config (#53843) (#69161)`` * ``Revert Dataform workflow invocation dict normalization (#69161) (#70395)`` * ``Prepare providers release 2026-07-22 (#70256)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` + * ``Prepare providers release 2026-08-01 (#70932)`` + * ``Update changelog with better wording (#71161)`` + * ``Prepare providers release 2026-08-06 (#71219)`` 22.2.2 ...... @@ -1438,7 +1445,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 13.0.0 ...... diff --git a/providers/google/docs/index.rst b/providers/google/docs/index.rst index 05c764f3c1cce..3234cbccc665d 100644 --- a/providers/google/docs/index.rst +++ b/providers/google/docs/index.rst @@ -110,9 +110,9 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================== PIP package Version required -========================================== ==================================================================== +========================================== ================================================================== ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.13.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` @@ -130,7 +130,14 @@ PIP package Version required ``google-auth`` ``>=2.29.0`` ``google-auth-httplib2`` ``>=0.0.1`` ``google-genai`` ``>=2.8.0`` -``google-cloud-aiplatform[evaluation]`` ``>=1.155.0`` +``google-cloud-aiplatform`` ``>=1.155.0`` +``tqdm`` ``>=4.23.0`` +``scikit-learn`` ``>=1.6.0`` +``jsonschema`` ``>=4.19.1`` +``ruamel.yaml`` ``>=0.19.0`` +``pyyaml`` ``>=6.0.3`` +``litellm`` ``>=1.83.7,<1.86.0; python_version < "3.14"`` +``litellm`` ``>=1.93.0; python_version >= "3.14"`` ``ray[default]`` ``>=2.42.0; python_version < "3.13"`` ``ray[default]`` ``>=2.49.0; python_version >= "3.13" and python_version < "3.14"`` ``ray[default]`` ``>=2.55.0; python_version >= "3.14" and python_version < "3.15"`` @@ -178,9 +185,9 @@ PIP package Version required ``httpx`` ``>=0.25.0`` ``looker-sdk`` ``>=22.4.0,!=24.18.0`` ``pandas-gbq`` ``>=0.7.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``proto-plus`` ``>=1.26.0`` ``pyarrow`` ``>=18.0.0; python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` @@ -191,7 +198,7 @@ PIP package Version required ``tenacity`` ``>=8.3.0`` ``immutabledict`` ``>=4.2.0`` ``types-protobuf`` ``>=5.27.0,!=5.29.1.20250402`` -========================================== ==================================================================== +========================================== ================================================================== Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/google/docs/secrets-backends/google-cloud-secret-manager-backend.rst b/providers/google/docs/secrets-backends/google-cloud-secret-manager-backend.rst index 7a4328ad3c233..091fa90a5bf4d 100644 --- a/providers/google/docs/secrets-backends/google-cloud-secret-manager-backend.rst +++ b/providers/google/docs/secrets-backends/google-cloud-secret-manager-backend.rst @@ -233,8 +233,12 @@ secret. Avoid ``--`` in connection ids and variable keys. .. warning:: - This refusal applies whether or not you use teams. If you already store a connection or - variable whose id contains ``--``, it stops resolving after upgrading and you must rename it. + This refusal only applies when ``[core] multi_team`` is enabled — with multi-team support + off, no team-scoped name can exist for a ``--``-containing id to collide with, so it resolves + normally. If you already store a connection or variable whose id contains ``--``, it keeps + working until multi-team support is enabled, at which point it silently stops resolving (a + warning is logged, but lookups do not raise) instead of raising outright. Check for ids + containing ``--`` before turning on ``multi_team``. ``get_config`` is not team-scoped and is unaffected by any of the above. diff --git a/providers/google/pyproject.toml b/providers/google/pyproject.toml index e750891904d1a..d4ae4fb1229fb 100644 --- a/providers/google/pyproject.toml +++ b/providers/google/pyproject.toml @@ -84,7 +84,19 @@ dependencies = [ # google-cloud-aiplatform doesn't install ray for python 3.12 (issue: https://github.com/googleapis/python-aiplatform/issues/5252). # Temporarily lock in ray 2.42.0 which is compatible with python 3.12 until linked issue is solved. # Remove the ray dependency as well as google-cloud-bigquery-storage once linked issue is fixed - "google-cloud-aiplatform[evaluation]>=1.155.0", + "google-cloud-aiplatform>=1.155.0", + # The "evaluation" extra of google-cloud-aiplatform limits litellm to range that has security + # vulnerabilities when we want to get 3.14 support. Until this is fixed we need to bump litellm to + # higher version and remove the "evaluation" extra from google-cloud-aiplatform. + # Tracked in: https://github.com/googleapis/python-aiplatform/issues/7057 + "tqdm>=4.23.0", + "scikit-learn>=1.6.0", + "jsonschema>=4.19.1", + "ruamel.yaml>=0.19.0", + "pyyaml>=6.0.3", + "litellm<1.86.0,>=1.83.7;python_version<'3.14'", + "litellm>=1.93.0;python_version>='3.14'", + # End of evolution extra workaround "ray[default]>=2.42.0;python_version<'3.13'", "ray[default]>=2.49.0;python_version>='3.13' and python_version <'3.14'", "ray[default]>=2.55.0;python_version>='3.14' and python_version <'3.15'", @@ -134,18 +146,9 @@ dependencies = [ # See https://github.com/looker-open-source/sdk-codegen/issues/1518 "looker-sdk>=22.4.0,!=24.18.0", "pandas-gbq>=0.7.0", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "proto-plus>=1.26.0", # Used to write parquet files by BaseSqlToGCSOperator "pyarrow>=18.0.0; python_version < '3.14'", diff --git a/providers/google/src/airflow/providers/google/ads/hooks/ads.py b/providers/google/src/airflow/providers/google/ads/hooks/ads.py index 806107dbbf082..786358625e988 100644 --- a/providers/google/src/airflow/providers/google/ads/hooks/ads.py +++ b/providers/google/src/airflow/providers/google/ads/hooks/ads.py @@ -31,10 +31,10 @@ from airflow.providers.google.common.hooks.base_google import get_field if TYPE_CHECKING: - from google.ads.googleads.v24.services.services.customer_service import CustomerServiceClient - from google.ads.googleads.v24.services.services.google_ads_service import GoogleAdsServiceClient - from google.ads.googleads.v24.services.services.google_ads_service.pagers import SearchPager - from google.ads.googleads.v24.services.types.google_ads_service import GoogleAdsRow + from collections.abc import Iterable + + import proto + from google.protobuf.message import Message as ProtobufMessage class GoogleAdsHook(BaseHook): @@ -111,7 +111,8 @@ class GoogleAdsHook(BaseHook): :param gcp_conn_id: The connection ID with the service account details. :param google_ads_conn_id: The connection ID with the details of Google Ads config.yaml file. - :param api_version: The Google Ads API version to use. + :param api_version: The Google Ads API version to use. If not set, the hook uses the default + version of the installed ``google-ads`` library. """ conn_name_attr = "google_ads_conn_id" @@ -162,7 +163,7 @@ def __init__( self.google_ads_config: dict[str, Any] = {} self.authentication_method: Literal["service_account", "developer_token"] = "service_account" - def search(self, client_ids: list[str], query: str, **kwargs) -> list[GoogleAdsRow]: + def search(self, client_ids: list[str], query: str, **kwargs) -> list[ProtobufMessage]: """ Pull data from the Google Ads API. @@ -185,7 +186,7 @@ def search(self, client_ids: list[str], query: str, **kwargs) -> list[GoogleAdsR return data_native_pb - def search_proto_plus(self, client_ids: list[str], query: str, **kwargs) -> list[GoogleAdsRow]: + def search_proto_plus(self, client_ids: list[str], query: str, **kwargs) -> list[proto.Message]: """ Pull data from the Google Ads API. @@ -226,9 +227,11 @@ def list_accessible_customers(self) -> list[str]: raise @cached_property - def _get_service(self) -> GoogleAdsServiceClient: + def _get_service(self) -> Any: """Connect and authenticate with the Google Ads API using a service account.""" client = self._get_client + if self.api_version is None: + return client.get_service("GoogleAdsService") return client.get_service("GoogleAdsService", version=self.api_version) @cached_property @@ -247,7 +250,7 @@ def _get_client(self) -> GoogleAdsClient: raise @cached_property - def _get_customer_service(self) -> CustomerServiceClient: + def _get_customer_service(self) -> Any: """Connect and authenticate with the Google Ads API using a service account.""" with NamedTemporaryFile("w", suffix=".json") as secrets_temp: self._get_config() @@ -256,6 +259,8 @@ def _get_customer_service(self) -> CustomerServiceClient: self._update_config_with_secret(secrets_temp) try: client = GoogleAdsClient.load_from_dict(self.google_ads_config) + if self.api_version is None: + return client.get_service("CustomerService") return client.get_service("CustomerService", version=self.api_version) except GoogleAuthError as e: self.log.error("Google Auth Error: %s", e) @@ -315,7 +320,7 @@ def _update_config_with_secret(self, secrets_temp: IO[str]) -> None: self.google_ads_config["json_key_file_path"] = secrets_temp.name - def _search(self, client_ids: list[str], query: str, **kwargs) -> list[GoogleAdsRow]: + def _search(self, client_ids: list[str], query: str, **kwargs) -> list[proto.Message]: """ Pull data from the Google Ads API. @@ -335,7 +340,7 @@ def _search(self, client_ids: list[str], query: str, **kwargs) -> list[GoogleAds return self._extract_rows(iterators) - def _extract_rows(self, iterators: list[SearchPager]) -> list[GoogleAdsRow]: + def _extract_rows(self, iterators: list[Iterable[proto.Message]]) -> list[proto.Message]: """ Convert Google Page Iterator (SearchPager) objects to Google Ads Rows. diff --git a/providers/google/src/airflow/providers/google/ads/operators/ads.py b/providers/google/src/airflow/providers/google/ads/operators/ads.py index 67ffb09b9d4ee..9848fe6d3c06a 100644 --- a/providers/google/src/airflow/providers/google/ads/operators/ads.py +++ b/providers/google/src/airflow/providers/google/ads/operators/ads.py @@ -62,7 +62,8 @@ class GoogleAdsListAccountsOperator(BaseOperator): If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated). - :param api_version: Optional Google Ads API version to use. + :param api_version: Optional Google Ads API version to use. If not set, the operator uses the + default version of the installed ``google-ads`` library. """ template_fields: Sequence[str] = ( diff --git a/providers/google/src/airflow/providers/google/ads/transfers/ads_to_gcs.py b/providers/google/src/airflow/providers/google/ads/transfers/ads_to_gcs.py index 041f13ec7c079..426ab5776be4d 100644 --- a/providers/google/src/airflow/providers/google/ads/transfers/ads_to_gcs.py +++ b/providers/google/src/airflow/providers/google/ads/transfers/ads_to_gcs.py @@ -61,7 +61,8 @@ class GoogleAdsToGcsOperator(BaseOperator): If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated). - :param api_version: Optional Google Ads API version to use. + :param api_version: Optional Google Ads API version to use. If not set, the operator uses the + default version of the installed ``google-ads`` library. :param unwrap_single: If True, return the GCS URI as a string instead of a list. Defaults to False for backward compatibility. """ diff --git a/providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py b/providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py index 8966e1372ae8c..8d62de670d820 100644 --- a/providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py +++ b/providers/google/src/airflow/providers/google/cloud/secrets/secret_manager.py @@ -23,7 +23,7 @@ from google.auth.exceptions import DefaultCredentialsError -from airflow.providers.common.compat.sdk import AirflowException +from airflow.providers.common.compat.sdk import AirflowException, conf from airflow.providers.google.cloud._internal_client.secret_manager_client import _SecretManagerClient from airflow.providers.google.cloud.utils.credentials_provider import ( _get_target_principal_and_delegates, @@ -248,7 +248,12 @@ def _names_a_team_namespace(self, secret_id: str) -> bool: backend does, is wrong here: the inherited implementation prepends a separator to an empty prefix (``'' -> '-smtp_default'``) and normalizes nothing, so the guard would both mis-anchor and miss ids whose separator only appears after normalization. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped secret can exist to collide with. """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False return TEAM_SEP in secret_id def _get_secret(self, path_prefix: str, secret_id: str, team_name: str | None = None) -> str | None: diff --git a/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/README.MD b/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/README.MD index f8be0c9191ec8..5137f46813706 100644 --- a/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/README.MD +++ b/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/README.MD @@ -1,6 +1,6 @@ # Dataflow Java Streaming -This project is a streaming example running with latest Apache Beam Java SDK (v.2.63), +This project is a streaming example running with latest Apache Beam Java SDK (v.2.75), as there is no "official" streaming example yet https://beam.apache.org/get-started/wordcount-example/. Sample logic to direct streaming data from input Pub/Sub source to the output PubSub topic to work @@ -11,7 +11,7 @@ That used for Java dataflow streaming system tests. ## Requirements -- **Java 17**: Ensure you have Java 17 installed and configured on your system. +- **Java 11 or higher**: Ensure you have Java installed and configured on your system - **Maven**: Make sure Maven is installed and configured on your system. - Maven is used for dependency management and building the project. diff --git a/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/pom.xml b/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/pom.xml index 85548bf7ab17e..4185bb1993394 100644 --- a/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/pom.xml +++ b/providers/google/tests/system/google/cloud/dataflow/resources/non_python_src/java_streaming_src/pom.xml @@ -26,16 +26,15 @@ v-0.1 UTF-8 - 2.63.0 - 2.0.16 - 1.113.0 + 2.75.0 + 2.0.18 + 1.152.0 2.45.0 2.45.0 - 17 - 17 + 11 - 3.8.1 - 3.2.4 + 3.15.0 + 3.6.2 diff --git a/providers/google/tests/unit/google/ads/hooks/test_ads.py b/providers/google/tests/unit/google/ads/hooks/test_ads.py index 1688ac83326ad..1e1f9727224bf 100644 --- a/providers/google/tests/unit/google/ads/hooks/test_ads.py +++ b/providers/google/tests/unit/google/ads/hooks/test_ads.py @@ -75,6 +75,14 @@ def mock_hook(request): yield hook +@pytest.fixture +def mock_default_version_hook(): + with mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") as conn: + hook = GoogleAdsHook() + conn.return_value.extra_dejson = EXTRAS_DEVELOPER_TOKEN + yield hook + + @pytest.fixture( params=[ {"input": EXTRAS_DEVELOPER_TOKEN, "expected_result": "developer_token"}, @@ -99,6 +107,12 @@ def test_get_customer_service(self, mock_client, mock_hook): client.assert_called_once_with(mock_hook.google_ads_config) client.return_value.get_service.assert_called_once_with("CustomerService", version=API_VERSION) + @mock.patch("airflow.providers.google.ads.hooks.ads.GoogleAdsClient") + def test_get_customer_service_uses_client_default_version(self, mock_client, mock_default_version_hook): + mock_default_version_hook._get_customer_service + + mock_client.load_from_dict.return_value.get_service.assert_called_once_with("CustomerService") + @mock.patch("airflow.providers.google.ads.hooks.ads.GoogleAdsClient") def test_get_service(self, mock_client, mock_hook): mock_hook._get_service() @@ -106,6 +120,12 @@ def test_get_service(self, mock_client, mock_hook): client.assert_called_once_with(mock_hook.google_ads_config) client.return_value.get_service.assert_called_once_with("GoogleAdsService", version=API_VERSION) + @mock.patch("airflow.providers.google.ads.hooks.ads.GoogleAdsClient") + def test_get_service_uses_client_default_version(self, mock_client, mock_default_version_hook): + mock_default_version_hook._get_service + + mock_client.load_from_dict.return_value.get_service.assert_called_once_with("GoogleAdsService") + @mock.patch("airflow.providers.google.ads.hooks.ads.GoogleAdsClient") def test_search(self, mock_client, mock_hook): service = mock_client.load_from_dict.return_value.get_service.return_value diff --git a/providers/google/tests/unit/google/cloud/secrets/test_secret_manager.py b/providers/google/tests/unit/google/cloud/secrets/test_secret_manager.py index 090f46b150e4d..0078bf7b300db 100644 --- a/providers/google/tests/unit/google/cloud/secrets/test_secret_manager.py +++ b/providers/google/tests/unit/google/cloud/secrets/test_secret_manager.py @@ -28,6 +28,10 @@ from airflow.providers.common.compat.sdk import AirflowException from airflow.providers.google.cloud.secrets.secret_manager import TEAM_SEP, CloudSecretManagerBackend +from tests_common.test_utils.config import conf_vars + +multi_team_enabled = conf_vars({("core", "multi_team"): "True"}) + CREDENTIALS = "test-creds" KEY_FILE = "test-file.json" PROJECT_ID = "test-project-id" @@ -318,6 +322,7 @@ def test_team_scoped_secret_is_resolved_for_its_own_team(self, mock_client, mock assert backend.get_conn_value(conn_id=CONN_ID, team_name=self.TEAM) == CONN_URI + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_team_scoped_secret_is_not_resolved_for_another_team(self, mock_client, mock_get_creds): @@ -329,6 +334,7 @@ def test_team_scoped_secret_is_not_resolved_for_another_team(self, mock_client, assert backend.get_conn_value(conn_id=encoded, team_name=self.OTHER_TEAM) is None + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_team_scoped_secret_is_not_resolved_without_a_team_scope(self, mock_client, mock_get_creds): @@ -339,6 +345,7 @@ def test_team_scoped_secret_is_not_resolved_without_a_team_scope(self, mock_clie assert backend.get_conn_value(conn_id=encoded) is None + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_team_whose_name_extends_the_callers_is_not_readable(self, mock_client, mock_get_creds): @@ -362,6 +369,7 @@ def test_team_agnostic_secret_is_resolved_for_any_team_scope(self, mock_client, assert backend.get_conn_value(conn_id=CONN_ID) == CONN_URI assert backend.get_conn_value(conn_id=CONN_ID, team_name=self.TEAM) == CONN_URI + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_team_scoped_variable_is_not_resolved_for_another_team(self, mock_client, mock_get_creds): @@ -416,6 +424,7 @@ def test_underscores_cannot_manufacture_a_team_namespace(self, mock_client, mock # the two names must not coincide in the first place assert backend._build_team_secret_name(CONNECTIONS_PREFIX, self.TEAM, f"prod__{CONN_ID}") != victim + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_ambiguous_id_is_refused_even_for_its_own_team(self, mock_client, mock_get_creds): @@ -432,6 +441,7 @@ def test_ambiguous_id_is_refused_even_for_its_own_team(self, mock_client, mock_g assert backend.get_conn_value(conn_id=ambiguous, team_name=self.TEAM) is None + @multi_team_enabled @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_refusing_an_ambiguous_id_is_logged(self, mock_client, mock_get_creds, caplog): @@ -448,6 +458,18 @@ def test_refusing_an_ambiguous_id_is_logged(self, mock_client, mock_get_creds, c assert all(r.levelname == "WARNING" for r in refusals) assert all(ambiguous in r.getMessage() for r in refusals) + @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") + @mock.patch(MODULE_NAME + "._SecretManagerClient") + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self, mock_client, mock_get_creds): + """No team scoped secret can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + mock_get_creds.return_value = CREDENTIALS, PROJECT_ID + backend, store = self._backend(mock_client) + ambiguous = f"prod{TEAM_SEP}{CONN_ID}" + store[backend.build_path(CONNECTIONS_PREFIX, ambiguous, SEP)] = CONN_URI + + assert backend.get_conn_value(conn_id=ambiguous) == CONN_URI + @mock.patch(MODULE_NAME + ".get_credentials_and_project_id") @mock.patch(MODULE_NAME + "._SecretManagerClient") def test_config_lookup_is_not_team_scoped(self, mock_client, mock_get_creds): diff --git a/providers/grpc/docs/changelog.rst b/providers/grpc/docs/changelog.rst index ca8ccc2fdedce..221fae7a44500 100644 --- a/providers/grpc/docs/changelog.rst +++ b/providers/grpc/docs/changelog.rst @@ -205,7 +205,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.7.1 ..... diff --git a/providers/http/docs/changelog.rst b/providers/http/docs/changelog.rst index 363553fa119bf..0a7bc0845cc3c 100644 --- a/providers/http/docs/changelog.rst +++ b/providers/http/docs/changelog.rst @@ -387,7 +387,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 5.1.0 ..... diff --git a/providers/imap/docs/changelog.rst b/providers/imap/docs/changelog.rst index 15d03a2d3013f..50e4a89299479 100644 --- a/providers/imap/docs/changelog.rst +++ b/providers/imap/docs/changelog.rst @@ -259,7 +259,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.8.1 ..... diff --git a/providers/influxdb/docs/changelog.rst b/providers/influxdb/docs/changelog.rst index d91daa7502cee..a6544fff3c96a 100644 --- a/providers/influxdb/docs/changelog.rst +++ b/providers/influxdb/docs/changelog.rst @@ -215,7 +215,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 2.8.1 ..... diff --git a/providers/jenkins/docs/changelog.rst b/providers/jenkins/docs/changelog.rst index 0851b21db55fc..3a6cb91a594f6 100644 --- a/providers/jenkins/docs/changelog.rst +++ b/providers/jenkins/docs/changelog.rst @@ -257,7 +257,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.0.1 ..... diff --git a/providers/keycloak/docs/changelog.rst b/providers/keycloak/docs/changelog.rst index 35f18bbee0b93..64f75516305fb 100644 --- a/providers/keycloak/docs/changelog.rst +++ b/providers/keycloak/docs/changelog.rst @@ -25,16 +25,26 @@ Changelog --------- +.. note:: + Upgrading the provider does not update existing Keycloak permissions. For each existing team, + run ``airflow keycloak-auth-manager create-team `` again to update the team ReadOnly + permission. For non-team installations, run + ``airflow keycloak-auth-manager create-permissions`` again without ``--teams`` to update the + global Admin permission. + + Manually added policies attached to these permissions will also be evaluated under the + ``AFFIRMATIVE`` strategy after the update. + 0.9.0 ..... .. note:: - Keycloak's access and refresh tokens are now stored in dedicated ``_access_token`` and - ``_refresh_token`` cookies instead of being carried inside the Airflow JWT claims. Sessions - established before this release carry the tokens in the old form, so the first request after - the upgrade cannot be refreshed and the session is cleared -- **every logged-in user is signed - out once when you upgrade**. No action is required beyond logging back in; this is a one-time - effect of the move and does not recur. + For Airflow v3.3.0+ Keycloak's access and refresh tokens are now stored in dedicated + ``_access_token`` and ``_refresh_token`` cookies instead of being carried inside the + Airflow JWT claims. Sessions established before this release carry the tokens in the old form, + so the first request after the upgrade cannot be refreshed and the session is cleared -- + **every logged-in user is signed out once when you upgrade**. No action is required + beyond logging back in; this is a one-time effect of the move and does not recur. Features ~~~~~~~~ diff --git a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/cli/commands.py b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/cli/commands.py index 705fc4eb0bfac..adb9aac6ff825 100644 --- a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/cli/commands.py +++ b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/cli/commands.py @@ -222,7 +222,7 @@ def _create_group_membership_mapper( "config": { "full.path": "false", "id.token.claim": "false", - "access.token.claim": "true", + "access.token.claim": "false", "userinfo.token.claim": "false", "claim.name": "groups", "jsonType.label": "String", @@ -301,6 +301,7 @@ def _attach_default_role_permissions( policy_name=_role_policy_name(role_name), scope_names=_get_extended_resource_methods() + ["LIST"], resource_names=[], + decision_strategy="AFFIRMATIVE", _dry_run=_dry_run, ) @@ -792,6 +793,7 @@ def _attach_team_permissions( policy_name=_team_role_policy_name(team, "Viewer"), scope_names=["GET", "LIST"], resource_names=team_readable_resources, + decision_strategy="AFFIRMATIVE", _dry_run=_dry_run, ) for role_name in ("User", "Op", "Admin"): @@ -802,6 +804,7 @@ def _attach_team_permissions( policy_name=_team_role_policy_name(team, role_name), scope_names=["GET", "LIST"], resource_names=team_readable_resources, + decision_strategy="AFFIRMATIVE", _dry_run=_dry_run, ) _attach_policy_to_scope_permission( @@ -926,6 +929,7 @@ def _attach_superadmin_permissions( policy_name=_role_policy_name(SUPER_ADMIN_ROLE_NAME), scope_names=_get_extended_resource_methods() + ["LIST"], resource_names=team_scoped_resources, + decision_strategy="AFFIRMATIVE", _dry_run=_dry_run, ) _attach_policy_to_scope_permission( diff --git a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/keycloak_auth_manager.py b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/keycloak_auth_manager.py index 1bb0c9bda1b17..ce0fc9338076a 100644 --- a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/keycloak_auth_manager.py +++ b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/keycloak_auth_manager.py @@ -64,6 +64,7 @@ ) from airflow.providers.keycloak.auth_manager.resources import KeycloakResource from airflow.providers.keycloak.auth_manager.user import KeycloakAuthManagerUser +from airflow.providers.keycloak.version_compat import AIRFLOW_V_3_3_PLUS from airflow.utils.helpers import prune_dict if TYPE_CHECKING: @@ -139,13 +140,24 @@ def http_session(self) -> requests.Session: def deserialize_user(self, token: dict[str, Any]) -> KeycloakAuthManagerUser: return KeycloakAuthManagerUser( - user_id=token["user_id"], name=token["name"], access_token="", refresh_token=None + user_id=token["user_id"], + name=token["name"], + access_token=token.get("access_token", ""), + refresh_token=token.get("refresh_token"), ) def serialize_user(self, user: KeycloakAuthManagerUser) -> dict[str, Any]: + if AIRFLOW_V_3_3_PLUS: + # Omit Keycloak JWTs from claims, they are stored in separate cookies + return { + "user_id": user.get_id(), + "name": user.get_name(), + } return { "user_id": user.get_id(), "name": user.get_name(), + "access_token": user.access_token, + "refresh_token": user.refresh_token, } async def get_user_from_token( @@ -159,6 +171,8 @@ async def get_user_from_token( :param refresh_token: Keycloak refresh JWT """ user = cast("KeycloakAuthManagerUser", await super().get_user_from_token(token)) + if not AIRFLOW_V_3_3_PLUS: + return user if access_token: user.access_token = access_token user.refresh_token = refresh_token diff --git a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/middleware.py b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/middleware.py index fcbfc56f044d3..1ad6f1eafa9de 100644 --- a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/middleware.py +++ b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/middleware.py @@ -78,8 +78,6 @@ async def dispatch(self, request: Request, call_next): user = new_user or current_user except ( AuthManagerRefreshTokenExpiredException, - ExpiredSignatureError, - InvalidTokenError, HTTPException, ): new_token = "" @@ -234,13 +232,17 @@ async def _refresh_user( access_token = request.cookies.get(COOKIE_NAME_ACCESS_TOKEN) refresh_token = request.cookies.get(COOKIE_NAME_REFRESH_TOKEN) if not jwt_token: - raise HTTPException( - status_code=status.HTTP_401_UNAUTHORIZED, detail="User is not logged into Airflow." - ) + # User is not logged into Airflow + return None, None if not access_token: raise HTTPException( status_code=status.HTTP_401_UNAUTHORIZED, detail="User is not logged into Keycloak." ) auth_manager = cast("KeycloakAuthManager", get_auth_manager()) - user = await auth_manager.get_user_from_token(jwt_token, access_token, refresh_token) + try: + user = await auth_manager.get_user_from_token(jwt_token, access_token, refresh_token) + except ExpiredSignatureError: + raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Token Expired") + except InvalidTokenError: + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid JWT token") return get_auth_manager().refresh_user(user=user), user diff --git a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/routes/login.py b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/routes/login.py index 804733757c4f2..0728518ab3c0f 100644 --- a/providers/keycloak/src/airflow/providers/keycloak/auth_manager/routes/login.py +++ b/providers/keycloak/src/airflow/providers/keycloak/auth_manager/routes/login.py @@ -28,7 +28,11 @@ from airflow.api_fastapi.app import AUTH_MANAGER_FASTAPI_APP_PREFIX, get_auth_manager from airflow.api_fastapi.auth.managers.base_auth_manager import COOKIE_NAME_JWT_TOKEN -from airflow.providers.keycloak.version_compat import AIRFLOW_V_3_1_1_PLUS, AIRFLOW_V_3_1_8_PLUS +from airflow.providers.keycloak.version_compat import ( + AIRFLOW_V_3_1_1_PLUS, + AIRFLOW_V_3_1_8_PLUS, + AIRFLOW_V_3_3_PLUS, +) if AIRFLOW_V_3_1_8_PLUS: from airflow.api_fastapi.app import get_cookie_path @@ -139,13 +143,14 @@ def login_callback(request: Request): COOKIE_NAME_ID_TOKEN, tokens["id_token"], path=cookie_path, secure=secure, httponly=True ) - response.set_cookie( - COOKIE_NAME_ACCESS_TOKEN, tokens["access_token"], path=cookie_path, secure=secure, httponly=True - ) + if AIRFLOW_V_3_3_PLUS: + response.set_cookie( + COOKIE_NAME_ACCESS_TOKEN, tokens["access_token"], path=cookie_path, secure=secure, httponly=True + ) - response.set_cookie( - COOKIE_NAME_REFRESH_TOKEN, tokens["refresh_token"], path=cookie_path, secure=secure, httponly=True - ) + response.set_cookie( + COOKIE_NAME_REFRESH_TOKEN, tokens["refresh_token"], path=cookie_path, secure=secure, httponly=True + ) return response @@ -197,16 +202,17 @@ def logout_callback(request: Request): secure=secure, httponly=True, ) - response.delete_cookie( - key=COOKIE_NAME_ACCESS_TOKEN, - path=cookie_path, - secure=secure, - httponly=True, - ) - response.delete_cookie( - key=COOKIE_NAME_REFRESH_TOKEN, - path=cookie_path, - secure=secure, - httponly=True, - ) + if AIRFLOW_V_3_3_PLUS: + response.delete_cookie( + key=COOKIE_NAME_ACCESS_TOKEN, + path=cookie_path, + secure=secure, + httponly=True, + ) + response.delete_cookie( + key=COOKIE_NAME_REFRESH_TOKEN, + path=cookie_path, + secure=secure, + httponly=True, + ) return response diff --git a/providers/keycloak/src/airflow/providers/keycloak/version_compat.py b/providers/keycloak/src/airflow/providers/keycloak/version_compat.py index e97df79b887d4..d87042a9edbb0 100644 --- a/providers/keycloak/src/airflow/providers/keycloak/version_compat.py +++ b/providers/keycloak/src/airflow/providers/keycloak/version_compat.py @@ -35,3 +35,4 @@ def get_base_airflow_version_tuple() -> tuple[int, int, int]: AIRFLOW_V_3_1_1_PLUS = get_base_airflow_version_tuple() >= (3, 1, 1) AIRFLOW_V_3_1_7_PLUS = get_base_airflow_version_tuple() >= (3, 1, 7) AIRFLOW_V_3_1_8_PLUS = get_base_airflow_version_tuple() >= (3, 1, 8) +AIRFLOW_V_3_3_PLUS = get_base_airflow_version_tuple() >= (3, 3, 0) diff --git a/providers/keycloak/tests/unit/keycloak/auth_manager/cli/test_commands.py b/providers/keycloak/tests/unit/keycloak/auth_manager/cli/test_commands.py index 62c61f95f7249..450996ad0a469 100644 --- a/providers/keycloak/tests/unit/keycloak/auth_manager/cli/test_commands.py +++ b/providers/keycloak/tests/unit/keycloak/auth_manager/cli/test_commands.py @@ -472,24 +472,17 @@ def test_create_permissions_attaches_default_role_policies( decision_strategy="AFFIRMATIVE", _dry_run=False, ) - mock_attach_scope_policy.assert_any_call( - client, - "test-id", - permission_name="Admin", - policy_name="Allow-Admin", - scope_names=_get_extended_resource_methods() + ["LIST"], - resource_names=[], - _dry_run=False, - ) - mock_attach_scope_policy.assert_any_call( - client, - "test-id", - permission_name="Admin", - policy_name="Allow-SuperAdmin", - scope_names=_get_extended_resource_methods() + ["LIST"], - resource_names=[], - _dry_run=False, - ) + for role_name in ("Admin", SUPER_ADMIN_ROLE_NAME): + mock_attach_scope_policy.assert_any_call( + client, + "test-id", + permission_name="Admin", + policy_name=f"Allow-{role_name}", + scope_names=_get_extended_resource_methods() + ["LIST"], + resource_names=[], + decision_strategy="AFFIRMATIVE", + _dry_run=False, + ) mock_attach_resource_policy.assert_any_call( client, "test-id", @@ -567,15 +560,17 @@ def test_create_team_command( mock_update_admin_permission_resources.assert_called_once_with(client, "test-id", _dry_run=False) mock_ensure_group_policy.assert_called_once_with(client, "test-id", "team-a", _dry_run=False) assert mock_ensure_aggregate_policy.call_count == 4 - mock_attach_policy.assert_any_call( - client, - "test-id", - permission_name="ReadOnly-team-a", - policy_name="Allow-Viewer-team-a", - scope_names=["GET", "LIST"], - resource_names=["Dag:team-a", "Team:team-a"], - _dry_run=False, - ) + for role_name in TEAM_ROLE_NAMES: + mock_attach_policy.assert_any_call( + client, + "test-id", + permission_name="ReadOnly-team-a", + policy_name=f"Allow-{role_name}-team-a", + scope_names=["GET", "LIST"], + resource_names=["Dag:team-a", "Team:team-a"], + decision_strategy="AFFIRMATIVE", + _dry_run=False, + ) mock_attach_policy.assert_any_call( client, "test-id", @@ -644,6 +639,7 @@ def test_create_team_command( "Team:team-a", "Variable:team-a", ], + decision_strategy="AFFIRMATIVE", _dry_run=False, ) mock_attach_policy.assert_any_call( diff --git a/providers/keycloak/tests/unit/keycloak/auth_manager/routes/test_login.py b/providers/keycloak/tests/unit/keycloak/auth_manager/routes/test_login.py index 058bda6590d89..dcfbb3202aebb 100644 --- a/providers/keycloak/tests/unit/keycloak/auth_manager/routes/test_login.py +++ b/providers/keycloak/tests/unit/keycloak/auth_manager/routes/test_login.py @@ -23,6 +23,7 @@ from airflow.api_fastapi.app import AUTH_MANAGER_FASTAPI_APP_PREFIX from tests_common.test_utils.config import conf_vars +from tests_common.test_utils.version_compat import AIRFLOW_V_3_3_PLUS class TestLoginRouter: @@ -79,8 +80,9 @@ def test_login_callback(self, mock_get_keycloak_client, mock_get_auth_manager, c assert "_token" in response.cookies assert response.cookies["_token"] == token assert response.cookies["_id_token"] == "id_token" - assert response.cookies["_access_token"] == "access_token" - assert response.cookies["_refresh_token"] == "refresh_token" + if AIRFLOW_V_3_3_PLUS: + assert response.cookies["_access_token"] == "access_token" + assert response.cookies["_refresh_token"] == "refresh_token" @patch("airflow.providers.keycloak.auth_manager.routes.login.KeycloakAuthManager.get_keycloak_client") def test_login_sets_secure_state_cookie_behind_tls_proxy(self, mock_get_keycloak_client, client): diff --git a/providers/keycloak/tests/unit/keycloak/auth_manager/test_keycloak_auth_manager.py b/providers/keycloak/tests/unit/keycloak/auth_manager/test_keycloak_auth_manager.py index 44ab60304bdae..d20c72e1aea5f 100644 --- a/providers/keycloak/tests/unit/keycloak/auth_manager/test_keycloak_auth_manager.py +++ b/providers/keycloak/tests/unit/keycloak/auth_manager/test_keycloak_auth_manager.py @@ -40,7 +40,11 @@ ) from tests_common.test_utils.config import conf_vars -from tests_common.test_utils.version_compat import AIRFLOW_V_3_1_7_PLUS, AIRFLOW_V_3_2_PLUS +from tests_common.test_utils.version_compat import ( + AIRFLOW_V_3_1_7_PLUS, + AIRFLOW_V_3_2_PLUS, + AIRFLOW_V_3_3_PLUS, +) if AIRFLOW_V_3_1_7_PLUS: from airflow.api_fastapi.auth.managers.exceptions import AuthManagerRefreshTokenExpiredException @@ -130,26 +134,47 @@ def _clear_filter_cache(): class TestKeycloakAuthManager: @pytest.mark.parametrize( - "token_data", + ("token_data", "exp"), [ - { - "user_id": "user_id", - "name": "name", - }, - { - "user_id": "user_id", - "name": "name", - "access_token": "access_token", - "refresh_token": "refresh_token", - }, + pytest.param( + { + "user_id": "user_id", + "name": "name", + }, + { + "user_id": "user_id", + "name": "name", + "access_token": "", + "refresh_token": None, + }, + marks=pytest.mark.skipif( + not AIRFLOW_V_3_3_PLUS, reason="Uses KeycloakJWTMiddleware and separate cookies" + ), + id="no-Keycloak-tokens", + ), + pytest.param( + { + "user_id": "user_id", + "name": "name", + "access_token": "access_token", + "refresh_token": "refresh_token", + }, + { + "user_id": "user_id", + "name": "name", + "access_token": "access_token", + "refresh_token": "refresh_token", + }, + id="with-Keycloak-tokens", + ), ], ) - def test_deserialize_user(self, auth_manager, token_data): + def test_deserialize_user(self, auth_manager, token_data, exp): result = auth_manager.deserialize_user(token_data) - assert result.user_id == "user_id" - assert result.name == "name" - assert result.access_token == "" - assert result.refresh_token is None + assert result.user_id == exp["user_id"] + assert result.name == exp["name"] + assert result.access_token == exp["access_token"] + assert result.refresh_token == exp["refresh_token"] def test_serialize_user(self, auth_manager): result = auth_manager.serialize_user( @@ -157,10 +182,19 @@ def test_serialize_user(self, auth_manager): user_id="user_id", name="name", access_token="access_token", refresh_token="refresh_token" ) ) - assert result == {"user_id": "user_id", "name": "name"} + if AIRFLOW_V_3_3_PLUS: + assert result == {"user_id": "user_id", "name": "name"} + else: + assert result == { + "user_id": "user_id", + "name": "name", + "access_token": "access_token", + "refresh_token": "refresh_token", + } + @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="Uses KeycloakJWTMiddleware and separate cookies") @pytest.mark.asyncio - async def test_get_user_from_token(self, auth_manager): + async def test_get_user_from_token_with_keycloak_tokens(self, auth_manager): mock_get_user_from_token = AsyncMock( return_value=KeycloakAuthManagerUser( user_id="user_id", name="name", access_token="", refresh_token=None @@ -180,6 +214,31 @@ async def test_get_user_from_token(self, auth_manager): assert user.access_token == "access_token" assert user.refresh_token == "refresh_token" + @pytest.mark.skipif(AIRFLOW_V_3_3_PLUS, reason="Testing Old Keycloak JWT flow.") + @pytest.mark.asyncio + async def test_get_user_from_token(self, auth_manager): + mock_token_validator = Mock() + mock_get_token_validator = Mock(return_value=mock_token_validator) + mock_token_validator.avalidated_claims = AsyncMock( + return_value=dict( + user_id="user_id", name="name", access_token="access_token", refresh_token="refresh_token" + ) + ) + with ( + patch.object( + KeycloakAuthManager, + "_get_token_validator", + mock_get_token_validator, + ), + ): + user = await auth_manager.get_user_from_token("token") + mock_token_validator.avalidated_claims.assert_called_with("token") + assert user.get_id() == "user_id" + assert user.get_name() == "name" + assert user.access_token == "access_token" + assert user.refresh_token == "refresh_token" + + @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="Uses KeycloakJWTMiddleware and separate cookies") @pytest.mark.asyncio async def test_get_user_from_token_keycloak_jwts_missing(self, auth_manager): mock_get_user_from_token = AsyncMock( @@ -196,6 +255,7 @@ async def test_get_user_from_token_keycloak_jwts_missing(self, auth_manager): ): assert await auth_manager.get_user_from_token("token") is None + @pytest.mark.skipif(not AIRFLOW_V_3_3_PLUS, reason="Uses KeycloakJWTMiddleware and separate cookies") @pytest.mark.asyncio async def test_get_user_from_token_keycloak_jwt(self, auth_manager): mock_get_user_from_token = AsyncMock( diff --git a/providers/keycloak/tests/unit/keycloak/auth_manager/test_middleware.py b/providers/keycloak/tests/unit/keycloak/auth_manager/test_middleware.py index 29a72988823d6..66f610ae30840 100644 --- a/providers/keycloak/tests/unit/keycloak/auth_manager/test_middleware.py +++ b/providers/keycloak/tests/unit/keycloak/auth_manager/test_middleware.py @@ -239,7 +239,7 @@ async def test_no_airflow_jwt_token( COOKIE_NAME_REFRESH_TOKEN: "refresh_token", } - await middleware.dispatch(mock_request, call_next) + response = await middleware.dispatch(mock_request, call_next) auth_manager.get_user_from_token.assert_not_called() auth_manager.refresh_user.assert_not_called() @@ -258,6 +258,7 @@ async def test_no_airflow_jwt_token( assert not hasattr(mock_request.state, "user_authenticated_via") call_next.assert_awaited_once_with(mock_request) + response.set_cookie.assert_not_called() @patch("airflow.providers.keycloak.auth_manager.middleware.get_auth_manager") @pytest.mark.asyncio diff --git a/providers/microsoft/azure/docs/changelog.rst b/providers/microsoft/azure/docs/changelog.rst index e7c9e0596446e..bed6667dca835 100644 --- a/providers/microsoft/azure/docs/changelog.rst +++ b/providers/microsoft/azure/docs/changelog.rst @@ -33,6 +33,7 @@ Changelog Features ~~~~~~~~ +* ``Add certificate-based authentication support to Microsoft Graph filesystem (#69335)`` * ``Add WasbRemoteLogIO.from_config and register wasb remote logging scheme (#70301)`` Bug Fixes @@ -42,6 +43,7 @@ Bug Fixes * ``Refuse the team agnostic fall-through for a team scoped Key Vault secret name (#70876)`` * ``Fix duplicated task logs in the WASB log handler (#70860)`` * ``Validate AzureVirtualMachineStateSensor target_state after rendering, outside poke() (#70329, #70372)`` +* ``Only refuse team scoped like secret ids when multi_team is on (#71078)`` Misc ~~~~ @@ -51,6 +53,7 @@ Misc .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): + * ``Fix some docs and test gaps following up multi team secret refusal (#71106)`` 14.0.0 ...... @@ -688,7 +691,7 @@ Misc ...... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 12.1.0 ...... diff --git a/providers/microsoft/azure/docs/filesystems/msgraph.rst b/providers/microsoft/azure/docs/filesystems/msgraph.rst index 7ebea783b7e80..90d8c5e40a759 100644 --- a/providers/microsoft/azure/docs/filesystems/msgraph.rst +++ b/providers/microsoft/azure/docs/filesystems/msgraph.rst @@ -35,13 +35,15 @@ Create a Microsoft Graph connection in Airflow with the following parameters: * **Connection Type**: msgraph * **Host**: Tenant ID * **Login**: Client ID -* **Password**: Client Secret +* **Password**: Client Secret, or certificate password when using certificate-based authentication The connection form provides additional configuration fields: * **Tenant ID**: Azure AD tenant identifier * **Drive ID**: Specific drive to access (optional - leave empty for general access) * **Scopes**: OAuth2 scopes (default: https://graph.microsoft.com/.default) +* **Certificate path**: File path to a PEM certificate for certificate-based authentication +* **Certificate data**: PEM certificate data for certificate-based authentication Additional OAuth2 parameters supported via connection extras: @@ -52,6 +54,10 @@ Additional OAuth2 parameters supported via connection extras: * **code_challenge_method**: PKCE code challenge method (e.g., 'S256') * **username**: Username for password grant flow * **password**: Password for password grant flow +* **certificate_path**: File path to a PEM certificate +* **certificate_data**: PEM certificate data +* **authority**: Microsoft Entra authority host +* **disable_instance_discovery**: Disable Microsoft Entra instance discovery Connection extra field configuration example: diff --git a/providers/microsoft/azure/src/airflow/providers/microsoft/azure/fs/msgraph.py b/providers/microsoft/azure/src/airflow/providers/microsoft/azure/fs/msgraph.py index f0988f961b6f9..b6fb11f76a323 100644 --- a/providers/microsoft/azure/src/airflow/providers/microsoft/azure/fs/msgraph.py +++ b/providers/microsoft/azure/src/airflow/providers/microsoft/azure/fs/msgraph.py @@ -26,6 +26,42 @@ from fsspec import AbstractFileSystem schemes = ["msgraph", "sharepoint", "onedrive", "msgd"] +DEFAULT_SCOPE = "https://graph.microsoft.com/.default" + + +def _get_token_endpoint(tenant_id: str) -> str: + return f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token" + + +def _get_scopes(options: dict[str, Any]) -> list[str]: + scopes = options.get("scope") or options.get("scopes") or DEFAULT_SCOPE + if isinstance(scopes, str): + return scopes.split() + return scopes + + +def _get_certificate_token(options: dict[str, Any]) -> dict[str, Any]: + from azure.identity import CertificateCredential + + credential = CertificateCredential( + tenant_id=options["tenant_id"], + client_id=options["client_id"], + password=options.get("password") or options.get("client_secret"), + certificate_path=options.get("certificate_path"), + certificate_data=options["certificate_data"].encode() if options.get("certificate_data") else None, + authority=options.get("authority"), + disable_instance_discovery=options.get("disable_instance_discovery", False), + ) + try: + access_token = credential.get_token(*_get_scopes(options)) + finally: + credential.close() + + return { + "access_token": access_token.token, + "token_type": "Bearer", + "expires_at": access_token.expires_on, + } def get_fs(conn_id: str | None, storage_options: dict[str, Any] | None = None) -> AbstractFileSystem: @@ -62,6 +98,7 @@ def get_fs(conn_id: str | None, storage_options: dict[str, Any] | None = None) - fields = [ "drive_id", "scope", + "scopes", "token_endpoint", "redirect_uri", "token_endpoint_auth_method", @@ -69,6 +106,10 @@ def get_fs(conn_id: str | None, storage_options: dict[str, Any] | None = None) - "update_token", "username", "password", + "certificate_path", + "certificate_data", + "authority", + "disable_instance_discovery", ] for field in fields: value = get_field(conn_id=conn_id, conn_type=conn_type, extras=extras, field_name=field) @@ -83,7 +124,19 @@ def get_fs(conn_id: str | None, storage_options: dict[str, Any] | None = None) - # Create oauth2 client parameters if authentication is provided oauth2_client_params = {} - if options.get("client_id") and options.get("client_secret") and options.get("tenant_id"): + if ( + options.get("client_id") + and options.get("tenant_id") + and (options.get("certificate_path") or options.get("certificate_data")) + ): + token_endpoint = options.get("token_endpoint") or _get_token_endpoint(options["tenant_id"]) + oauth2_client_params = { + "client_id": options["client_id"], + "token": _get_certificate_token(options), + "token_endpoint": token_endpoint, + "scope": " ".join(_get_scopes(options)), + } + elif options.get("client_id") and options.get("client_secret") and options.get("tenant_id"): oauth2_client_params = { "client_id": options["client_id"], "client_secret": options["client_secret"], @@ -105,11 +158,12 @@ def get_fs(conn_id: str | None, storage_options: dict[str, Any] | None = None) - if param in options: oauth2_client_params[param] = options[param] + if "scopes" in options and "scope" not in oauth2_client_params: + oauth2_client_params["scope"] = " ".join(_get_scopes(options)) + # Construct default token_endpoint from tenant_id if not explicitly provided - if "token_endpoint" not in oauth2_client_params and tenant_id: - oauth2_client_params["token_endpoint"] = ( - f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token" - ) + if "token_endpoint" not in oauth2_client_params: + oauth2_client_params["token_endpoint"] = _get_token_endpoint(options["tenant_id"]) # Determine which filesystem to return based on drive_id drive_id = options.get("drive_id") diff --git a/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py b/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py index 7162663ca1dfe..30a20a827e6b5 100644 --- a/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py +++ b/providers/microsoft/azure/src/airflow/providers/microsoft/azure/secrets/key_vault.py @@ -33,6 +33,7 @@ from azure.identity import ClientSecretCredential, DefaultAzureCredential from azure.keyvault.secrets import SecretClient +from airflow.providers.common.compat.sdk import conf from airflow.providers.microsoft.azure.utils import get_sync_default_azure_credential from airflow.secrets import BaseSecretsBackend from airflow.utils.log.logging_mixin import LoggingMixin @@ -247,7 +248,12 @@ def _names_a_team_namespace(self, secret_id: str) -> bool: The id is normalised first because :meth:`build_path` maps ``_`` onto the separator everywhere in this backend, so ``b__c`` reaches Key Vault as ``b--c`` and would otherwise manufacture the team separator from an id that does not visibly contain it. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped secret can exist to collide with. """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False return TEAM_SEP in self.build_path("", secret_id, self.sep) def _log_refusal(self, kind: str, secret_id: str) -> None: diff --git a/providers/microsoft/azure/tests/unit/microsoft/azure/fs/test_msgraph.py b/providers/microsoft/azure/tests/unit/microsoft/azure/fs/test_msgraph.py index d95a772dda296..92a2142275275 100644 --- a/providers/microsoft/azure/tests/unit/microsoft/azure/fs/test_msgraph.py +++ b/providers/microsoft/azure/tests/unit/microsoft/azure/fs/test_msgraph.py @@ -137,14 +137,18 @@ def test_get_fs_with_storage_options(self, mock_msgdrivefs, mock_get_connection, mock_fs_instance = MagicMock() mock_msgdrivefs.return_value = mock_fs_instance - storage_options = {"drive_id": "storage_drive_id", "scope": "custom.scope"} + storage_options = { + "drive_id": "storage_drive_id", + "scope": "custom.scope", + "tenant_id": "storage_tenant_id", + } result = get_fs("msgraph_minimal", storage_options=storage_options) expected_oauth2_params = { "client_id": "test_client_id", "client_secret": "test_client_secret", - "tenant_id": "test_tenant_id", - "token_endpoint": "https://login.microsoftonline.com/test_tenant_id/oauth2/v2.0/token", + "tenant_id": "storage_tenant_id", + "token_endpoint": "https://login.microsoftonline.com/storage_tenant_id/oauth2/v2.0/token", "scope": "custom.scope", } mock_msgdrivefs.assert_called_once_with( @@ -152,6 +156,102 @@ def test_get_fs_with_storage_options(self, mock_msgdrivefs, mock_get_connection, ) assert result == mock_fs_instance + @patch("azure.identity.CertificateCredential", autospec=True) + @patch("airflow.providers.microsoft.azure.fs.msgraph.BaseHook.get_connection", autospec=True) + @patch("msgraphfs.MSGDriveFS", autospec=True) + def test_get_fs_with_certificate_path( + self, mock_msgdrivefs, mock_get_connection, mock_certificate_credential + ): + connection = Connection( + conn_id="msgraph_certificate", + conn_type="msgraph", + login="test_client_id", + password="certificate_password", + host="test_tenant_id", + extra={"drive_id": "test_drive_id", "certificate_path": "/tmp/cert.pem"}, + ) + mock_get_connection.return_value = connection + mock_fs_instance = MagicMock() + mock_msgdrivefs.return_value = mock_fs_instance + mock_access_token = MagicMock(token="certificate-token", expires_on=1234567890) + mock_certificate_credential.return_value.get_token.return_value = mock_access_token + + result = get_fs("msgraph_certificate") + + mock_certificate_credential.assert_called_once_with( + tenant_id="test_tenant_id", + client_id="test_client_id", + password="certificate_password", + certificate_path="/tmp/cert.pem", + certificate_data=None, + authority=None, + disable_instance_discovery=False, + ) + mock_certificate_credential.return_value.get_token.assert_called_once_with( + "https://graph.microsoft.com/.default" + ) + mock_certificate_credential.return_value.close.assert_called_once_with() + mock_msgdrivefs.assert_called_once_with( + drive_id="test_drive_id", + oauth2_client_params={ + "client_id": "test_client_id", + "token": { + "access_token": "certificate-token", + "token_type": "Bearer", + "expires_at": 1234567890, + }, + "token_endpoint": "https://login.microsoftonline.com/test_tenant_id/oauth2/v2.0/token", + "scope": "https://graph.microsoft.com/.default", + }, + ) + assert result == mock_fs_instance + + @patch("azure.identity.CertificateCredential", autospec=True) + @patch("airflow.providers.microsoft.azure.fs.msgraph.BaseHook.get_connection", autospec=True) + @patch("msgraphfs.MSGDriveFS", autospec=True) + def test_get_fs_with_certificate_data_from_storage_options( + self, mock_msgdrivefs, mock_get_connection, mock_certificate_credential, mock_connection_minimal + ): + mock_get_connection.return_value = mock_connection_minimal + mock_fs_instance = MagicMock() + mock_msgdrivefs.return_value = mock_fs_instance + mock_access_token = MagicMock(token="storage-token", expires_on=1234567890) + mock_certificate_credential.return_value.get_token.return_value = mock_access_token + + result = get_fs( + "msgraph_minimal", + storage_options={ + "certificate_data": "certificate-data", + "scope": "custom.scope", + "token_endpoint": "https://login.microsoftonline.com/custom/oauth2/v2.0/token", + }, + ) + + mock_certificate_credential.assert_called_once_with( + tenant_id="test_tenant_id", + client_id="test_client_id", + password="test_client_secret", + certificate_path=None, + certificate_data=b"certificate-data", + authority=None, + disable_instance_discovery=False, + ) + mock_certificate_credential.return_value.get_token.assert_called_once_with("custom.scope") + mock_msgdrivefs.assert_called_once_with( + drive_id=None, + oauth2_client_params={ + "client_id": "test_client_id", + "token": { + "access_token": "storage-token", + "token_type": "Bearer", + "expires_at": 1234567890, + }, + "token_endpoint": "https://login.microsoftonline.com/custom/oauth2/v2.0/token", + "scope": "custom.scope", + }, + ) + assert result == mock_fs_instance + @patch("airflow.providers.microsoft.azure.fs.msgraph.BaseHook.get_connection") @patch("msgraphfs.MSGDriveFS") def test_get_fs_incomplete_credentials(self, mock_msgdrivefs, mock_get_connection): diff --git a/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py b/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py index b3f44d2299c64..26903998acd89 100644 --- a/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py +++ b/providers/microsoft/azure/tests/unit/microsoft/azure/secrets/test_key_vault.py @@ -24,7 +24,10 @@ from airflow.providers.microsoft.azure.secrets.key_vault import AzureKeyVaultBackend +from tests_common.test_utils.config import conf_vars + KEY_VAULT_MODULE = "airflow.providers.microsoft.azure.secrets.key_vault" +multi_team_enabled = conf_vars({("core", "multi_team"): "True"}) class TestAzureKeyVaultBackend: @@ -107,6 +110,7 @@ def test_get_variable_uses_team_secret_with_custom_prefix(self, mock_client): assert secret_val == "team-value" mock_client.get_secret.assert_called_once_with(name="custom-variables-team-a--hello") + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_get_variable_returns_none_for_team_scoped_key_without_team_name(self, mock_client): backend = AzureKeyVaultBackend() @@ -114,6 +118,7 @@ def test_get_variable_returns_none_for_team_scoped_key_without_team_name(self, m assert backend.get_variable("teama--hello") is None mock_client.get_secret.assert_not_called() + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_another_teams_secret_is_not_reachable(self, mock_client): """A caller scoped to one team must not reach another team's secret by naming it. @@ -133,6 +138,7 @@ def only_the_target_exists(name): assert backend.get_conn_value("teama--my_db", team_name="teamb") is None + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_team_whose_name_extends_the_callers_is_not_reachable(self, mock_client): """A prefix match on the caller's own namespace is not proof of ownership. @@ -153,6 +159,7 @@ def only_the_target_exists(name): assert backend.get_conn_value("teama--prod--my_db", team_name="teama") is None + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace(self, mock_client): """The team scoped name is not safe by construction -- the id can extend it. @@ -174,6 +181,7 @@ def only_the_target_exists(name): assert backend.get_conn_value("prod--my_db", team_name="teama") is None + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_underscores_cannot_manufacture_a_team_namespace(self, mock_client): """``build_path`` maps ``_`` onto the separator, so ``__`` becomes the team separator. @@ -193,6 +201,7 @@ def only_the_target_exists(name): assert backend.get_conn_value("prod__my_db", team_name="teama") is None + @multi_team_enabled @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") def test_refusing_an_ambiguous_id_is_logged(self, mock_client, caplog): """A silent ``None`` is indistinguishable from a missing secret, so the refusal is logged.""" @@ -215,6 +224,21 @@ def test_refusing_an_ambiguous_id_is_logged(self, mock_client, caplog): assert sum(refused_id in r.getMessage() for r in refusals) == 1 mock_client.get_secret.assert_not_called() + @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend.client") + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self, mock_client): + """No team scoped secret can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + mock_client.get_secret.return_value = mock.Mock(value="world") + backend = AzureKeyVaultBackend() + + assert backend.get_conn_value("prod--my_db") == "world" + assert backend.get_variable("prod__hello") == "world" + assert backend.get_config("prod--sql_alchemy_conn") == "world" + + mock_client.get_secret.assert_any_call(name="airflow-connections-prod--my-db") + mock_client.get_secret.assert_any_call(name="airflow-variables-prod--hello") + mock_client.get_secret.assert_any_call(name="airflow-config-prod--sql-alchemy-conn") + @mock.patch(f"{KEY_VAULT_MODULE}.AzureKeyVaultBackend._get_secret") def test_variable_prefix_none_value(self, mock_get_secret): """ diff --git a/providers/microsoft/mssql/docs/changelog.rst b/providers/microsoft/mssql/docs/changelog.rst index 08a69b7af5388..827d934a98e54 100644 --- a/providers/microsoft/mssql/docs/changelog.rst +++ b/providers/microsoft/mssql/docs/changelog.rst @@ -245,7 +245,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 4.1.0 ..... diff --git a/providers/microsoft/winrm/docs/changelog.rst b/providers/microsoft/winrm/docs/changelog.rst index f9449cf77f7ab..acc66fd232d4e 100644 --- a/providers/microsoft/winrm/docs/changelog.rst +++ b/providers/microsoft/winrm/docs/changelog.rst @@ -288,7 +288,7 @@ Misc ..... .. note:: - This version has no code changes. It's released due to yank of previous version due to packaging issues. + This version contains no code changes. It was released to replace a previous version that was yanked due to a packaging issue. 3.8.0 ..... diff --git a/providers/opensearch/README.rst b/providers/opensearch/README.rst index 2f12cdaa3d6fe..43361067bb585 100644 --- a/providers/opensearch/README.rst +++ b/providers/opensearch/README.rst @@ -23,7 +23,7 @@ Package ``apache-airflow-providers-opensearch`` -Release: ``1.11.2`` +Release: ``1.12.0`` `OpenSearch `__ @@ -36,7 +36,7 @@ This is a provider package for ``opensearch`` provider. All classes for this pro are in ``airflow.providers.opensearch`` python package. You can find package information and changelog for the provider -in the `documentation `_. +in the `documentation `_. Installation ------------ @@ -59,4 +59,4 @@ PIP package Version required ========================================== ================== The changelog for the provider package can be found in the -`changelog `_. +`changelog `_. diff --git a/providers/opensearch/docs/changelog.rst b/providers/opensearch/docs/changelog.rst index 1f8553c1ddc4e..5941538ccd637 100644 --- a/providers/opensearch/docs/changelog.rst +++ b/providers/opensearch/docs/changelog.rst @@ -27,9 +27,14 @@ Changelog --------- -1.11.2 +1.12.0 ...... +Features +~~~~~~~~ + +* ``Add OpensearchRemoteLogIO.from_config and register opensearch scheme (#70295)`` + Bug Fixes ~~~~~~~~~ diff --git a/providers/opensearch/docs/index.rst b/providers/opensearch/docs/index.rst index b142444e48348..84b157f4b515b 100644 --- a/providers/opensearch/docs/index.rst +++ b/providers/opensearch/docs/index.rst @@ -78,7 +78,7 @@ apache-airflow-providers-opensearch package `OpenSearch `__ -Release: 1.11.2 +Release: 1.12.0 Provider package ---------------- @@ -112,5 +112,5 @@ Downloading official packages You can download officially released packages and verify their checksums and signatures from the `Official Apache Download site `_ -* `The apache-airflow-providers-opensearch 1.11.2 sdist package `_ (`asc `__, `sha512 `__) -* `The apache-airflow-providers-opensearch 1.11.2 wheel package `_ (`asc `__, `sha512 `__) +* `The apache-airflow-providers-opensearch 1.12.0 sdist package `_ (`asc `__, `sha512 `__) +* `The apache-airflow-providers-opensearch 1.12.0 wheel package `_ (`asc `__, `sha512 `__) diff --git a/providers/opensearch/docs/logging/index.rst b/providers/opensearch/docs/logging/index.rst index 43f849217110c..f64aee61b772f 100644 --- a/providers/opensearch/docs/logging/index.rst +++ b/providers/opensearch/docs/logging/index.rst @@ -44,6 +44,22 @@ First, to use the handler, ``airflow.cfg`` must be configured as follows: username = password = +On Airflow 3.3.0 or above you can also route remote logging to OpenSearch through the +provider dispatch mechanism by adding an ``opensearch://`` scheme to +``[logging] remote_base_log_folder``: + +.. code-block:: ini + + [logging] + remote_logging = True + remote_base_log_folder = opensearch:// + + [opensearch] + host = + port = + username = + password = + To output task logs to stdout in JSON format, the following config could be used: .. code-block:: ini diff --git a/providers/opensearch/provider.yaml b/providers/opensearch/provider.yaml index e0d5fed38ba40..24b8bef46cc78 100644 --- a/providers/opensearch/provider.yaml +++ b/providers/opensearch/provider.yaml @@ -23,13 +23,13 @@ description: | state: ready lifecycle: production -source-date-epoch: 1785633505 +source-date-epoch: 1785995745 # Note that those versions are maintained by release manager - do not update them manually # with the exception of case where other provider in sources has >= new provider version. # In such case adding >= NEW_VERSION and bumping to NEW_VERSION in a provider have # to be done in the same PR versions: - - 1.11.2 + - 1.12.0 - 1.11.1 - 1.11.0 - 1.10.0 @@ -96,6 +96,10 @@ connection-types: logging: - airflow.providers.opensearch.log.os_task_handler.OpensearchTaskHandler +remote-logging: + - classpath: airflow.providers.opensearch.log.os_task_handler.OpensearchRemoteLogIO + scheme: opensearch + config: opensearch: description: ~ diff --git a/providers/opensearch/pyproject.toml b/providers/opensearch/pyproject.toml index 2d49f24eb3dd7..eb956a48663e6 100644 --- a/providers/opensearch/pyproject.toml +++ b/providers/opensearch/pyproject.toml @@ -25,7 +25,7 @@ build-backend = "flit_core.buildapi" [project] name = "apache-airflow-providers-opensearch" -version = "1.11.2" +version = "1.12.0" description = "Provider package apache-airflow-providers-opensearch for Apache Airflow" readme = "README.rst" license = "Apache-2.0" @@ -99,8 +99,8 @@ apache-airflow-providers-common-sql = {workspace = true} apache-airflow-providers-standard = {workspace = true} [project.urls] -"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-opensearch/1.11.2" -"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-opensearch/1.11.2/changelog.html" +"Documentation" = "https://airflow.apache.org/docs/apache-airflow-providers-opensearch/1.12.0" +"Changelog" = "https://airflow.apache.org/docs/apache-airflow-providers-opensearch/1.12.0/changelog.html" "Bug Tracker" = "https://github.com/apache/airflow/issues" "Source Code" = "https://github.com/apache/airflow" "Slack Chat" = "https://s.apache.org/airflow-slack" diff --git a/providers/opensearch/src/airflow/providers/opensearch/__init__.py b/providers/opensearch/src/airflow/providers/opensearch/__init__.py index d63f5cd2e73c6..9cb402530b37c 100644 --- a/providers/opensearch/src/airflow/providers/opensearch/__init__.py +++ b/providers/opensearch/src/airflow/providers/opensearch/__init__.py @@ -29,7 +29,7 @@ __all__ = ["__version__"] -__version__ = "1.11.2" +__version__ = "1.12.0" if packaging.version.parse(packaging.version.parse(airflow_version).base_version) < packaging.version.parse( "3.0.0" diff --git a/providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py b/providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py index 70a3fac8ec291..c6a0368e2e94b 100644 --- a/providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py +++ b/providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py @@ -60,6 +60,12 @@ def get_provider_info(): } ], "logging": ["airflow.providers.opensearch.log.os_task_handler.OpensearchTaskHandler"], + "remote-logging": [ + { + "classpath": "airflow.providers.opensearch.log.os_task_handler.OpensearchRemoteLogIO", + "scheme": "opensearch", + } + ], "config": { "opensearch": { "description": None, diff --git a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py b/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py index 6d9722478a5db..2ad4db39daa23 100644 --- a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py +++ b/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py @@ -873,6 +873,44 @@ class OpensearchRemoteLogIO(LoggingMixin): # noqa: D101 processors = () + @classmethod + def from_config(cls) -> OpensearchRemoteLogIO: + """ + Build the remote log IO from Airflow logging and ``[opensearch]`` configuration. + + The ``opensearch://`` value in ``[logging] remote_base_log_folder`` is only a routing + marker, so every connection and behaviour parameter is read from the ``[opensearch]`` + section here. + + This does not merge ``[logging] remote_task_handler_kwargs`` IO-kwargs, matching the + legacy behavior for OpenSearch. + """ + remote_task_handler_kwargs = conf.getjson("logging", "remote_task_handler_kwargs", fallback={}) + if not isinstance(remote_task_handler_kwargs, dict): + raise ValueError( + "logging/remote_task_handler_kwargs must be a JSON object (a python dict), we got " + f"{type(remote_task_handler_kwargs)}" + ) + # ``[opensearch] port`` declares an empty-string default, so the key is always present and + # ``conf.getint`` raises on ``int("")`` instead of falling back to 9200. + port = conf.get("opensearch", "port", fallback="") + return cls( + base_log_folder=os.path.expanduser(conf.get_mandatory_value("logging", "base_log_folder")), + delete_local_copy=conf.getboolean("logging", "delete_local_logs"), + host=conf.get("opensearch", "host", fallback=""), + port=int(port) if port else 9200, + username=conf.get_mandatory_value("opensearch", "username"), + password=conf.get_mandatory_value("opensearch", "password"), + write_stdout=conf.getboolean("opensearch", "write_stdout"), + write_to_opensearch=conf.getboolean("opensearch", "write_to_os"), + json_format=conf.getboolean("opensearch", "json_format"), + target_index=conf.get_mandatory_value("opensearch", "target_index"), + host_field=conf.get_mandatory_value("opensearch", "host_field"), + offset_field=conf.get_mandatory_value("opensearch", "offset_field"), + log_id_template=conf.get("opensearch", "log_id_template", fallback="") + or "{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}", + ) + def __attrs_post_init__(self): self.host = _format_url(self.host) self.port = self.port if self.port is not None else (urlparse(self.host).port or 9200) diff --git a/providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py b/providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py index ca49103b8a5e5..c877ab5cbf11c 100644 --- a/providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py +++ b/providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py @@ -20,6 +20,7 @@ import dataclasses import json import logging +import os import re from io import StringIO from pathlib import Path @@ -786,6 +787,126 @@ def test_upload_returns_early_when_ti_is_none(self, tmp_path): self.opensearch_io.upload(log_file, ti=None) +class TestOpensearchRemoteLogIOFromConfig: + @conf_vars( + { + ("logging", "base_log_folder"): "~/airflow/logs", + ("logging", "delete_local_logs"): "True", + ("opensearch", "host"): "https://opensearch.example.com:9200", + ("opensearch", "port"): "9201", + ("opensearch", "username"): "admin", + ("opensearch", "password"): "secret", + ("opensearch", "write_stdout"): "True", + ("opensearch", "write_to_os"): "True", + ("opensearch", "json_format"): "True", + ("opensearch", "target_index"): "my-logs", + ("opensearch", "host_field"): "host.name", + ("opensearch", "offset_field"): "log.offset", + ("opensearch", "log_id_template"): "{dag_id}-{task_id}-{run_id}", + } + ) + def test_from_config(self): + subject = OpensearchRemoteLogIO.from_config() + + assert subject.base_log_folder == Path(os.path.expanduser("~/airflow/logs")) + assert subject.delete_local_copy is True + assert subject.host == "https://opensearch.example.com:9200" + assert subject.port == 9201 + assert subject.username == "admin" + assert subject.password == "secret" + assert subject.write_stdout is True + assert subject.write_to_opensearch is True + assert subject.json_format is True + assert subject.target_index == "my-logs" + assert subject.host_field == "host.name" + assert subject.offset_field == "log.offset" + assert subject.log_id_template == "{dag_id}-{task_id}-{run_id}" + + @conf_vars( + { + ("logging", "base_log_folder"): "/tmp/airflow/logs", + ("logging", "delete_local_logs"): "False", + ("opensearch", "host"): "https://opensearch.example.com:9200", + ("opensearch", "username"): "admin", + ("opensearch", "password"): "secret", + ("logging", "remote_task_handler_kwargs"): '{"delete_local_copy": true, "max_bytes": 1024}', + } + ) + def test_from_config_ignores_remote_task_handler_kwargs(self): + """Unlike the object-storage backends, OpenSearch does not merge IO kwargs (legacy parity).""" + subject = OpensearchRemoteLogIO.from_config() + + # ``delete_local_copy`` stays at the ``[logging] delete_local_logs`` value. + assert subject.delete_local_copy is False + # ``max_bytes`` belongs to FileTaskHandler and must not reach the IO class. + assert not hasattr(subject, "max_bytes") + + @conf_vars( + { + ("logging", "base_log_folder"): "/tmp/airflow/logs", + ("opensearch", "host"): "https://opensearch.example.com:9201", + ("opensearch", "port"): "", + ("opensearch", "username"): "admin", + ("opensearch", "password"): "secret", + } + ) + def test_from_config_defaults_port_when_unset(self): + """An unset port falls back to 9200 (legacy intent) rather than the host URL's port.""" + subject = OpensearchRemoteLogIO.from_config() + + assert subject.port == 9200 + + @conf_vars({("logging", "remote_task_handler_kwargs"): '["not", "a", "dict"]'}) + def test_from_config_rejects_non_dict_remote_task_handler_kwargs(self): + with pytest.raises(ValueError, match="remote_task_handler_kwargs"): + OpensearchRemoteLogIO.from_config() + + def test_provider_registers_opensearch_scheme(self): + from airflow.providers_manager import ProvidersManager + + manager = ProvidersManager() + if not hasattr(manager, "remote_logging_handler_by_scheme"): + pytest.skip("Airflow core does not support remote logging provider dispatch") + + info = manager.remote_logging_handler_by_scheme("opensearch") + + assert info is not None + assert info.classpath == "airflow.providers.opensearch.log.os_task_handler.OpensearchRemoteLogIO" + + @pytest.mark.parametrize( + "manager_classpath", + [ + pytest.param("airflow.providers_manager.ProvidersManager", id="core"), + pytest.param( + "airflow.sdk.providers_manager_runtime.ProvidersManagerTaskRuntime", id="task-runtime" + ), + ], + ) + @conf_vars( + { + ("logging", "remote_logging"): "True", + ("logging", "remote_base_log_folder"): "opensearch://", + ("opensearch", "host"): "https://opensearch.example.com:9200", + ("opensearch", "username"): "admin", + ("opensearch", "password"): "secret", + } + ) + def test_resolve_remote_task_log_uses_provider_dispatch_not_local_settings(self, manager_classpath): + factory = pytest.importorskip("airflow._shared.logging.factory") + from airflow._shared.module_loading import import_string + from airflow.configuration import conf + + with patch.object(factory, "discover_remote_log_handler", autospec=True) as legacy_discover: + remote_task_log, _ = factory.resolve_remote_task_log( + conf=conf, + providers_manager=import_string(manager_classpath)(), + import_string=import_string, + ) + + assert isinstance(remote_task_log, OpensearchRemoteLogIO) + legacy_discover.assert_not_called() + + class TestFormatErrorDetail: def test_returns_none_for_empty(self): assert _format_error_detail(None) is None diff --git a/providers/papermill/README.rst b/providers/papermill/README.rst index f53de9e48e91b..6e7a189fa6e14 100644 --- a/providers/papermill/README.rst +++ b/providers/papermill/README.rst @@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.8.0`` ``papermill[all]`` ``>=2.6.0`` ``scrapbook[all]`` ``>=0.5.0`` ``ipykernel`` ``>=6.29.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``nbconvert`` ``>=7.16.1`` -========================================== ==================================================================== +========================================== ================================================================= The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/papermill/docs/changelog.rst b/providers/papermill/docs/changelog.rst index 172b01923bf61..805f5b31b8c69 100644 --- a/providers/papermill/docs/changelog.rst +++ b/providers/papermill/docs/changelog.rst @@ -33,7 +33,6 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Fix template-field validation in PapermillOperator (#70435)`` .. Below changes are excluded from the changelog. Move them to @@ -42,6 +41,8 @@ Misc * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` * ``Prepare provider documentation 2026-06-16 (#68642)`` * ``[main] Upgrade important CI environment (#68322)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 3.13.1 ...... diff --git a/providers/papermill/docs/index.rst b/providers/papermill/docs/index.rst index 84db1c1016f5c..4a00ba47673eb 100644 --- a/providers/papermill/docs/index.rst +++ b/providers/papermill/docs/index.rst @@ -97,19 +97,19 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.8.0`` ``papermill[all]`` ``>=2.6.0`` ``scrapbook[all]`` ``>=0.5.0`` ``ipykernel`` ``>=6.29.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``nbconvert`` ``>=7.16.1`` -========================================== ==================================================================== +========================================== ================================================================= Downloading official packages ----------------------------- diff --git a/providers/papermill/pyproject.toml b/providers/papermill/pyproject.toml index 8351b32b79944..cfe221235f792 100644 --- a/providers/papermill/pyproject.toml +++ b/providers/papermill/pyproject.toml @@ -64,18 +64,9 @@ dependencies = [ "papermill[all]>=2.6.0", "scrapbook[all]>=0.5.0", "ipykernel>=6.29.4", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "nbconvert>=7.16.1", ] diff --git a/providers/postgres/README.rst b/providers/postgres/README.rst index ce30e20880f3b..30f6fec670c94 100644 --- a/providers/postgres/README.rst +++ b/providers/postgres/README.rst @@ -87,18 +87,18 @@ Dependent package Optional dependencies ---------------------- -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ Extra Dependencies -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ ``amazon`` ``apache-airflow-providers-amazon>=2.6.0`` ``asyncpg`` ``asyncpg>=0.30.0`` ``microsoft.azure`` ``apache-airflow-providers-microsoft-azure>=12.8.0`` ``openlineage`` ``apache-airflow-providers-openlineage`` -``pandas`` ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``polars`` ``polars>=1.26.0`` ``psycopg2`` ``psycopg2-binary>=2.9.9; python_version < '3.13'``, ``psycopg2-binary>=2.9.10; python_version >= '3.13'`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/postgres/docs/changelog.rst b/providers/postgres/docs/changelog.rst index cb23caa096ad9..c056cf37a9ce9 100644 --- a/providers/postgres/docs/changelog.rst +++ b/providers/postgres/docs/changelog.rst @@ -33,7 +33,6 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Use built-in exceptions for Postgres hook input validation (#70537)`` Doc-only @@ -43,6 +42,8 @@ Doc-only .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 7.0.0 ..... diff --git a/providers/postgres/docs/index.rst b/providers/postgres/docs/index.rst index afcb301214219..56e5152bdc8c4 100644 --- a/providers/postgres/docs/index.rst +++ b/providers/postgres/docs/index.rst @@ -143,18 +143,18 @@ Install them when installing from PyPI. For example: pip install apache-airflow-providers-postgres[amazon] -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ Extra Dependencies -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ ``amazon`` ``apache-airflow-providers-amazon>=2.6.0`` ``asyncpg`` ``asyncpg>=0.30.0`` ``microsoft.azure`` ``apache-airflow-providers-microsoft-azure>=12.8.0`` ``openlineage`` ``apache-airflow-providers-openlineage`` -``pandas`` ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"`` +``pandas`` ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"`` ``polars`` ``polars>=1.26.0`` ``psycopg2`` ``psycopg2-binary>=2.9.9; python_version < '3.13'``, ``psycopg2-binary>=2.9.10; python_version >= '3.13'`` ``sqlalchemy`` ``sqlalchemy>=1.4.54`` -=================== ===================================================================================================================================================================== +=================== ============================================================================================================================================================ Downloading official packages ----------------------------- diff --git a/providers/postgres/pyproject.toml b/providers/postgres/pyproject.toml index 583f29b2cdcfe..d95164ae27f7f 100644 --- a/providers/postgres/pyproject.toml +++ b/providers/postgres/pyproject.toml @@ -93,18 +93,9 @@ dependencies = [ "apache-airflow-providers-openlineage" ] "pandas" = [ - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] "polars" = [ "polars>=1.26.0" diff --git a/providers/presto/README.rst b/providers/presto/README.rst index 4aea9bdc9490b..f9486643f1374 100644 --- a/providers/presto/README.rst +++ b/providers/presto/README.rst @@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``presto-python-client`` ``>=0.8.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``psycopg2-binary`` ``>=2.9.9; python_version < "3.13"`` ``psycopg2-binary`` ``>=2.9.10; python_version >= "3.13"`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/presto/docs/changelog.rst b/providers/presto/docs/changelog.rst index b70570f676a47..50cba487ca7d2 100644 --- a/providers/presto/docs/changelog.rst +++ b/providers/presto/docs/changelog.rst @@ -33,7 +33,6 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` Doc-only ~~~~~~~~ @@ -44,6 +43,8 @@ Doc-only appropriate section above if needed. Do not delete the lines(!): * ``Prepare providers release 2026-07-22 (#70256)`` * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 5.12.0 ...... diff --git a/providers/presto/docs/index.rst b/providers/presto/docs/index.rst index b990a5da459a0..65317d259a88f 100644 --- a/providers/presto/docs/index.rst +++ b/providers/presto/docs/index.rst @@ -98,19 +98,19 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` ``presto-python-client`` ``>=0.8.4`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``psycopg2-binary`` ``>=2.9.9; python_version < "3.13"`` ``psycopg2-binary`` ``>=2.9.10; python_version >= "3.13"`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/presto/pyproject.toml b/providers/presto/pyproject.toml index c1fde9da0d943..cb1506fe164fb 100644 --- a/providers/presto/pyproject.toml +++ b/providers/presto/pyproject.toml @@ -63,18 +63,9 @@ dependencies = [ "apache-airflow-providers-common-compat>=1.12.0", "apache-airflow-providers-common-sql>=1.32.0", "presto-python-client>=0.8.4", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', 'psycopg2-binary>=2.9.9; python_version < "3.13"', 'psycopg2-binary>=2.9.10; python_version >= "3.13"', ] diff --git a/providers/salesforce/README.rst b/providers/salesforce/README.rst index ab5ca3e0a15b8..a6455cdb169b0 100644 --- a/providers/salesforce/README.rst +++ b/providers/salesforce/README.rst @@ -50,16 +50,16 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.10.1`` ``simple-salesforce`` ``>=1.0.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/salesforce/docs/changelog.rst b/providers/salesforce/docs/changelog.rst index 69fc4e6d7383a..24d2cf703cd4a 100644 --- a/providers/salesforce/docs/changelog.rst +++ b/providers/salesforce/docs/changelog.rst @@ -33,11 +33,13 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` + * ``Add support for pandas 3 based xcoms in airflow (#71103)`` 5.14.1 ...... diff --git a/providers/salesforce/docs/index.rst b/providers/salesforce/docs/index.rst index 2fa5706cbe949..69ab6cdf6ca7f 100644 --- a/providers/salesforce/docs/index.rst +++ b/providers/salesforce/docs/index.rst @@ -97,16 +97,16 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.10.1`` ``simple-salesforce`` ``>=1.0.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= Downloading official packages ----------------------------- diff --git a/providers/salesforce/pyproject.toml b/providers/salesforce/pyproject.toml index 38780a67cc7ab..0908820c2f83c 100644 --- a/providers/salesforce/pyproject.toml +++ b/providers/salesforce/pyproject.toml @@ -62,18 +62,9 @@ dependencies = [ "apache-airflow>=2.11.0", "apache-airflow-providers-common-compat>=1.10.1", "simple-salesforce>=1.0.0", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] [dependency-groups] diff --git a/providers/salesforce/tests/unit/salesforce/hooks/test_salesforce.py b/providers/salesforce/tests/unit/salesforce/hooks/test_salesforce.py index 39e2e41fde6eb..923d59a39569e 100644 --- a/providers/salesforce/tests/unit/salesforce/hooks/test_salesforce.py +++ b/providers/salesforce/tests/unit/salesforce/hooks/test_salesforce.py @@ -24,12 +24,15 @@ import numpy as np import pandas as pd import pytest +from packaging.version import Version from requests import Session as request_session from simple_salesforce import Salesforce, api from airflow.models.connection import Connection from airflow.providers.salesforce.hooks.salesforce import SalesforceHook +PANDAS_3_PLUS = Version(pd.__version__).major >= 3 + class TestSalesforceHook: def setup_method(self): @@ -349,10 +352,12 @@ def test_write_object_to_file_csv(self, mock_data_frame): data_frame = self.salesforce_hook.write_object_to_file(query_results=[], filename=filename, fmt="csv") mock_data_frame.return_value.to_csv.assert_called_once_with(filename, index=False) - # Note that the latest version of pandas dataframes (1.1.2) returns "nan" rather than "None" here + # Note that the latest version of pandas dataframes (1.1.2) returns "nan" rather than "None" here, + # and pandas 3 keeps missing values as NA instead of stringifying them to "nan" at all + missing = np.nan if PANDAS_3_PLUS else "nan" pd.testing.assert_frame_equal( data_frame, - pd.DataFrame({"test": [1, 2, 3], "dict": ["nan", "nan", str({"foo": "bar"})]}), + pd.DataFrame({"test": [1, 2, 3], "dict": [missing, missing, str({"foo": "bar"})]}), check_index_type=False, ) diff --git a/providers/samba/docs/index.rst b/providers/samba/docs/index.rst index 63ff1e5fb6fbc..1241d03f31f64 100644 --- a/providers/samba/docs/index.rst +++ b/providers/samba/docs/index.rst @@ -134,11 +134,12 @@ Install them when installing from PyPI. For example: pip install apache-airflow-providers-samba[google] -========== =================================== -Extra Dependencies -========== =================================== -``google`` ``apache-airflow-providers-google`` -========== =================================== +============ ================================================= +Extra Dependencies +============ ================================================= +``google`` ``apache-airflow-providers-google`` +``kerberos`` ``krb5>=0.8.0``, ``smbprotocol[kerberos]>=1.5.0`` +============ ================================================= Downloading official packages ----------------------------- diff --git a/providers/samba/provider.yaml b/providers/samba/provider.yaml index 17cbc0e02430f..564051413e29d 100644 --- a/providers/samba/provider.yaml +++ b/providers/samba/provider.yaml @@ -98,6 +98,20 @@ connection-types: description: >- The share OS type (`posix` or `windows`). Used to determine the formatting of file and folder paths. + auth_protocol: + label: Auth Protocol + schema: + type: + - string + - 'null' + default: negotiate + enum: + - negotiate + - ntlm + - kerberos + description: >- + Must be one of: negotiate, ntlm, kerberos. With `kerberos`, the system's + ticket cache is used and login/password are optional. ui-field-behaviour: hidden-fields: [] relabeling: diff --git a/providers/samba/pyproject.toml b/providers/samba/pyproject.toml index 268199698f8e1..36dc9b2b6d92f 100644 --- a/providers/samba/pyproject.toml +++ b/providers/samba/pyproject.toml @@ -70,6 +70,10 @@ dependencies = [ "google" = [ "apache-airflow-providers-google" ] +"kerberos" = [ + "krb5>=0.8.0", + "smbprotocol[kerberos]>=1.5.0", +] [dependency-groups] dev = [ diff --git a/providers/samba/src/airflow/providers/samba/get_provider_info.py b/providers/samba/src/airflow/providers/samba/get_provider_info.py index fa6f98566f7b8..4f57b7a9ebcda 100644 --- a/providers/samba/src/airflow/providers/samba/get_provider_info.py +++ b/providers/samba/src/airflow/providers/samba/get_provider_info.py @@ -53,7 +53,16 @@ def get_provider_info(): "label": "Share Type", "schema": {"type": ["string", "null"], "default": "posix"}, "description": "The share OS type (`posix` or `windows`). Used to determine the formatting of file and folder paths.", - } + }, + "auth_protocol": { + "label": "Auth Protocol", + "schema": { + "type": ["string", "null"], + "default": "negotiate", + "enum": ["negotiate", "ntlm", "kerberos"], + }, + "description": "Must be one of: negotiate, ntlm, kerberos. With `kerberos`, the system's ticket cache is used and login/password are optional.", + }, }, "ui-field-behaviour": {"hidden-fields": [], "relabeling": {"schema": "Share"}}, } diff --git a/providers/samba/src/airflow/providers/samba/hooks/samba.py b/providers/samba/src/airflow/providers/samba/hooks/samba.py index c36f0b06dc929..1e6bf5a9cded8 100644 --- a/providers/samba/src/airflow/providers/samba/hooks/samba.py +++ b/providers/samba/src/airflow/providers/samba/hooks/samba.py @@ -24,7 +24,7 @@ import smbclient -from airflow.providers.common.compat.sdk import BaseHook +from airflow.providers.common.compat.sdk import AirflowOptionalProviderFeatureException, BaseHook if TYPE_CHECKING: import smbprotocol.connection @@ -43,6 +43,8 @@ class SambaHook(BaseHook): the connection is used in its place. :param share_type: An optional share type name. If this is unset then it will assume a posix share type. + :param auth_protocol: + An optional authentication protocol. If this is unset then it defaults to negotiate. """ conn_name_attr = "samba_conn_id" @@ -50,22 +52,50 @@ class SambaHook(BaseHook): conn_type = "samba" hook_name = "Samba" + VALID_AUTH_PROTOCOLS = {"negotiate", "ntlm", "kerberos"} + def __init__( self, samba_conn_id: str = default_conn_name, share: str | None = None, share_type: Literal["posix", "windows"] | None = None, + auth_protocol: Literal["negotiate", "ntlm", "kerberos"] | None = None, ) -> None: super().__init__() conn = self.get_connection(samba_conn_id) + extra = conn.extra_dejson + + legacy_auth = extra.get("auth") + legacy_auth_protocol = ( + legacy_auth + if isinstance(legacy_auth, str) and legacy_auth in self.VALID_AUTH_PROTOCOLS + else "negotiate" + ) + self._auth_protocol: str = auth_protocol or extra.get("auth_protocol", legacy_auth_protocol) + if self._auth_protocol not in self.VALID_AUTH_PROTOCOLS: + raise ValueError( + f"Invalid auth_protocol '{self._auth_protocol}'. " + f"Must be one of {sorted(self.VALID_AUTH_PROTOCOLS)}." + ) - if not conn.login: + uses_kerberos = self._auth_protocol == "kerberos" + + if uses_kerberos: + try: + import krb5 # noqa: F401 + except ImportError: + raise AirflowOptionalProviderFeatureException( + "Kerberos authentication requires the 'krb5' package. " + "Install it with: pip install 'apache-airflow-providers-samba[kerberos]'" + ) + + if not conn.login and not uses_kerberos: self.log.info("Login not provided") - if not conn.password: + if not conn.password and not uses_kerberos: self.log.info("Password not provided") - self._share_type = share_type or conn.extra_dejson.get("share_type", "posix") + self._share_type = share_type or extra.get("share_type", "posix") if self._share_type not in {"posix", "windows"}: self._share_type = "posix" self.log.warning( @@ -77,11 +107,12 @@ def __init__( self._host = conn.host self._share = share or conn.schema self._connection_cache = connection_cache - self._conn_kwargs = { + self._conn_kwargs: dict[str, Any] = { "username": conn.login, "password": conn.password, "port": conn.port or 445, "connection_cache": connection_cache, + "auth_protocol": self._auth_protocol, } def __enter__(self): @@ -325,12 +356,22 @@ def get_ui_field_behaviour(cls) -> dict[str, Any]: def get_connection_form_widgets(cls) -> dict[str, Any]: """Return connection widgets to add to connection form.""" from flask_babel import lazy_gettext - from wtforms import StringField + from wtforms import SelectField, StringField return { "share_type": StringField( label=lazy_gettext("Share Type"), description="The share OS type (`posix` or `windows`). Used to determine the formatting of file and folder paths.", default="posix", - ) + ), + "auth_protocol": SelectField( + label=lazy_gettext("Auth Protocol"), + description=( + "Authentication protocol: `negotiate` (auto-select, default), " + "`ntlm`, or `kerberos`. When using `kerberos`, the system's " + "Kerberos ticket cache is used and username/password are optional." + ), + choices=["negotiate", "ntlm", "kerberos"], + default="negotiate", + ), } diff --git a/providers/samba/tests/unit/samba/hooks/test_samba.py b/providers/samba/tests/unit/samba/hooks/test_samba.py index 44d79c8830353..06a5e3fb147eb 100644 --- a/providers/samba/tests/unit/samba/hooks/test_samba.py +++ b/providers/samba/tests/unit/samba/hooks/test_samba.py @@ -23,7 +23,10 @@ import pytest from airflow.models import Connection -from airflow.providers.common.compat.sdk import AirflowNotFoundException +from airflow.providers.common.compat.sdk import ( + AirflowNotFoundException, + AirflowOptionalProviderFeatureException, +) from airflow.providers.samba.hooks.samba import SambaHook try: @@ -63,6 +66,7 @@ def test_context_manager(self, get_conn_mock, register_session): "password": CONNECTION.password, "port": 445, "connection_cache": {}, + "auth_protocol": "negotiate", } cache = kwargs.get("connection_cache") mock_connection = mock.Mock() @@ -117,6 +121,7 @@ def test_method(self, get_conn_mock, name): "username": CONNECTION.login, "password": CONNECTION.password, "port": 445, + "auth_protocol": "negotiate", } with mock.patch("smbclient." + name) as p: kwargs = {} @@ -191,6 +196,116 @@ def test__join_path( hook = SambaHook("samba_default", share_type=path_type) assert hook._join_path(path) == full_path + @mock.patch.dict("sys.modules", {"krb5": mock.MagicMock()}) + @mock.patch("smbclient.register_session") + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_kerberos_auth_via_extra(self, get_conn_mock, register_session): + """Test that auth_protocol='kerberos' from extra is passed to smbclient.""" + connection = Connection( + host="kerb-host.example.com", + schema="share", + extra='{"auth_protocol": "kerberos"}', + ) + get_conn_mock.return_value = connection + register_session.return_value = None + with SambaHook("samba_default"): + _, kwargs = tuple(register_session.call_args_list[0]) + assert kwargs["auth_protocol"] == "kerberos" + assert kwargs["username"] is None + assert kwargs["password"] is None + + @mock.patch.dict("sys.modules", {"krb5": mock.MagicMock()}) + @mock.patch("smbclient.register_session") + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_kerberos_auth_via_legacy_auth_key(self, get_conn_mock, register_session): + """Test backward compat: extra {"auth": "kerberos"} is recognized.""" + connection = Connection( + host="kerb-host.example.com", + schema="share", + extra='{"auth": "kerberos"}', + ) + get_conn_mock.return_value = connection + register_session.return_value = None + with SambaHook("samba_default"): + _, kwargs = tuple(register_session.call_args_list[0]) + assert kwargs["auth_protocol"] == "kerberos" + + @mock.patch.dict("sys.modules", {"krb5": mock.MagicMock()}) + @mock.patch("smbclient.register_session") + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_kerberos_auth_via_constructor(self, get_conn_mock, register_session): + """Test that constructor auth_protocol overrides extra.""" + connection = Connection( + host="kerb-host.example.com", + schema="share", + login="user", + password="pass", + ) + get_conn_mock.return_value = connection + register_session.return_value = None + with SambaHook("samba_default", auth_protocol="kerberos"): + _, kwargs = tuple(register_session.call_args_list[0]) + assert kwargs["auth_protocol"] == "kerberos" + + @mock.patch("smbclient.register_session") + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_legacy_auth_key_ignored_when_invalid(self, get_conn_mock, register_session): + """Test that extra {"auth": "basic"} is ignored and defaults to negotiate.""" + connection = Connection( + host="host", + schema="share", + login="user", + password="pass", + extra='{"auth": "basic"}', + ) + get_conn_mock.return_value = connection + register_session.return_value = None + with SambaHook("samba_default"): + _, kwargs = tuple(register_session.call_args_list[0]) + assert kwargs["auth_protocol"] == "negotiate" + + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_invalid_auth_protocol_raises(self, get_conn_mock): + """Test that an invalid auth_protocol raises ValueError.""" + connection = Connection( + host="host", + schema="share", + extra='{"auth_protocol": "invalid"}', + ) + get_conn_mock.return_value = connection + with pytest.raises(ValueError, match="Invalid auth_protocol 'invalid'"): + SambaHook("samba_default") + + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_kerberos_without_dependency_raises(self, get_conn_mock): + """Test that kerberos auth without krb5 installed raises AirflowOptionalProviderFeatureException.""" + connection = Connection( + host="kerb-host.example.com", + schema="share", + extra='{"auth_protocol": "kerberos"}', + ) + get_conn_mock.return_value = connection + with mock.patch.dict("sys.modules", {"krb5": None}): + with pytest.raises(AirflowOptionalProviderFeatureException, match="krb5"): + SambaHook("samba_default") + + @mock.patch("smbclient.register_session") + @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") + def test_ntlm_auth(self, get_conn_mock, register_session): + """Test that auth_protocol='ntlm' is passed correctly.""" + connection = Connection( + host="host", + schema="share", + login="user", + password="pass", + extra='{"auth_protocol": "ntlm"}', + ) + get_conn_mock.return_value = connection + register_session.return_value = None + with SambaHook("samba_default"): + _, kwargs = tuple(register_session.call_args_list[0]) + assert kwargs["auth_protocol"] == "ntlm" + @mock.patch("airflow.providers.samba.hooks.samba.smbclient.open_file", return_value=mock.Mock()) @mock.patch(f"{BASEHOOK_PATCH_PATH}.get_connection") def test_open_file(self, get_conn_mock, open_file_mock): diff --git a/providers/snowflake/README.rst b/providers/snowflake/README.rst index 680cb0d8cf374..f79d395a8d180 100644 --- a/providers/snowflake/README.rst +++ b/providers/snowflake/README.rst @@ -56,12 +56,13 @@ PIP package Version required ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyarrow`` ``>=16.1.0; python_version < "3.13"`` ``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` +``pyOpenSSL`` ``>=22.1.0`` ``snowflake-connector-python`` ``>=3.17.0`` ``snowflake-sqlalchemy`` ``>=1.7.0`` ``snowflake-snowpark-python`` ``>=1.17.0,<9999; python_version < "3.12"`` diff --git a/providers/snowflake/docs/changelog.rst b/providers/snowflake/docs/changelog.rst index 26878d8fb8e35..fce8b438052bb 100644 --- a/providers/snowflake/docs/changelog.rst +++ b/providers/snowflake/docs/changelog.rst @@ -39,11 +39,11 @@ Bug Fixes ~~~~~~~~~ * ``Log check_query_output response via format string in Snowflake SQL API hook (#67848)`` +* ``Fix snowflake provider installs picking an unimportable pyOpenSSL (#71145)`` Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` * ``Use common.compat.sdk for timezone imports in providers (#70492)`` * ``Move the SnowparkContainerJobOperator required-argument check to execute() (#70332, #70874)`` @@ -54,6 +54,8 @@ Doc-only .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 6.15.0 ...... diff --git a/providers/snowflake/docs/index.rst b/providers/snowflake/docs/index.rst index 1fb9aa7c2ec7a..66299a9ea7e04 100644 --- a/providers/snowflake/docs/index.rst +++ b/providers/snowflake/docs/index.rst @@ -105,12 +105,13 @@ PIP package Version required ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``pyarrow`` ``>=16.1.0; python_version < "3.13"`` ``pyarrow`` ``>=18.0.0; python_version >= "3.13" and python_version < "3.14"`` ``pyarrow`` ``>=22.0.0; python_version >= "3.14"`` +``pyOpenSSL`` ``>=22.1.0`` ``snowflake-connector-python`` ``>=3.17.0`` ``snowflake-sqlalchemy`` ``>=1.7.0`` ``snowflake-snowpark-python`` ``>=1.17.0,<9999; python_version < "3.12"`` diff --git a/providers/snowflake/pyproject.toml b/providers/snowflake/pyproject.toml index 5503d6e48cc3f..1f3ca1a782d46 100644 --- a/providers/snowflake/pyproject.toml +++ b/providers/snowflake/pyproject.toml @@ -62,21 +62,16 @@ dependencies = [ "apache-airflow>=2.11.0", "apache-airflow-providers-common-compat>=1.12.0", "apache-airflow-providers-common-sql>=1.32.0", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "pyarrow>=16.1.0; python_version < '3.13'", "pyarrow>=18.0.0; python_version >= '3.13' and python_version < '3.14'", "pyarrow>=22.0.0; python_version >= '3.14'", + # snowflake-connector-python <4.4.0 caps pyOpenSSL below 26.0.0, and pyOpenSSL 22.0.0 and older + # declare no upper bound on cryptography, so an unconstrained resolve pairs them with a + # cryptography they cannot import against. 22.1.0 is the first release with a correct bound. + "pyOpenSSL>=22.1.0", # TODO(potiuk): We should bump the snowflake-connector-python to >=4.0.0 when sqlalchemy>=2.0 is required # 3.17.0 is the first version with Workload Identity Federation (WORKLOAD_IDENTITY authenticator). "snowflake-connector-python>=3.17.0", diff --git a/providers/trino/README.rst b/providers/trino/README.rst index 2575de229574b..59a497c0bb4e3 100644 --- a/providers/trino/README.rst +++ b/providers/trino/README.rst @@ -50,17 +50,17 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``trino`` ``>=0.319.0`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/trino/docs/changelog.rst b/providers/trino/docs/changelog.rst index d3742cc310a9b..c7da9c63b1321 100644 --- a/providers/trino/docs/changelog.rst +++ b/providers/trino/docs/changelog.rst @@ -32,7 +32,6 @@ Changelog Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` Doc-only ~~~~~~~~ @@ -43,6 +42,8 @@ Doc-only appropriate section above if needed. Do not delete the lines(!): * ``Prepare providers release 2026-07-22 (#70256)`` * ``Fix inconsistency between generated provider docs and pyproject.toml (#68991)`` + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 6.6.0 ..... diff --git a/providers/trino/docs/index.rst b/providers/trino/docs/index.rst index aefcf7db66c98..079a0d94a50b1 100644 --- a/providers/trino/docs/index.rst +++ b/providers/trino/docs/index.rst @@ -98,17 +98,17 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.12.0`` ``apache-airflow-providers-common-sql`` ``>=1.32.0`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` ``trino`` ``>=0.319.0`` -========================================== ==================================================================== +========================================== ================================================================= Optional cross provider package dependencies -------------------------------------------- diff --git a/providers/trino/pyproject.toml b/providers/trino/pyproject.toml index 3f72a5f242b9d..b0e66c047c878 100644 --- a/providers/trino/pyproject.toml +++ b/providers/trino/pyproject.toml @@ -62,18 +62,9 @@ dependencies = [ "apache-airflow>=2.11.0", "apache-airflow-providers-common-compat>=1.12.0", "apache-airflow-providers-common-sql>=1.32.0", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', "trino>=0.319.0", ] diff --git a/providers/weaviate/README.rst b/providers/weaviate/README.rst index 593ad8725db5e..d4198503860e0 100644 --- a/providers/weaviate/README.rst +++ b/providers/weaviate/README.rst @@ -50,17 +50,17 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14 Requirements ------------ -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.8.0`` ``httpx`` ``>=0.25.0`` ``weaviate-client`` ``>=4.16.0,!=4.16.7`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= The changelog for the provider package can be found in the `changelog `_. diff --git a/providers/weaviate/docs/changelog.rst b/providers/weaviate/docs/changelog.rst index 81a39ec50bf40..0da137969b21d 100644 --- a/providers/weaviate/docs/changelog.rst +++ b/providers/weaviate/docs/changelog.rst @@ -31,7 +31,6 @@ Bug Fixes Misc ~~~~ -* ``Limit pandas to < 3 for DataFrame XComs (#70791)`` Doc-only ~~~~~~~~ @@ -40,6 +39,8 @@ Doc-only .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): + * ``Limit pandas to < 3 for DataFrame XComs (#70791)`` + * ``Revert "Limit pandas to < 3 for DataFrame XComs (#70791)" (#71100)`` 3.4.0 ..... diff --git a/providers/weaviate/docs/index.rst b/providers/weaviate/docs/index.rst index 1b5e52e66506f..48b8b058e9199 100644 --- a/providers/weaviate/docs/index.rst +++ b/providers/weaviate/docs/index.rst @@ -99,17 +99,17 @@ Requirements The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``. -========================================== ==================================================================== +========================================== ================================================================= PIP package Version required -========================================== ==================================================================== +========================================== ================================================================= ``apache-airflow`` ``>=2.11.0`` ``apache-airflow-providers-common-compat`` ``>=1.8.0`` ``httpx`` ``>=0.25.0`` ``weaviate-client`` ``>=4.16.0,!=4.16.7`` -``pandas`` ``>=2.1.2,<3; python_version < "3.13"`` -``pandas`` ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"`` -``pandas`` ``>=2.3.3,<3; python_version >= "3.14"`` -========================================== ==================================================================== +``pandas`` ``>=2.1.2; python_version < "3.13"`` +``pandas`` ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"`` +``pandas`` ``>=2.3.3; python_version >= "3.14"`` +========================================== ================================================================= Downloading official packages ----------------------------- diff --git a/providers/weaviate/pyproject.toml b/providers/weaviate/pyproject.toml index 7c1aaefe14aa4..02d1adfc0926b 100644 --- a/providers/weaviate/pyproject.toml +++ b/providers/weaviate/pyproject.toml @@ -63,18 +63,9 @@ dependencies = [ "apache-airflow-providers-common-compat>=1.8.0", "httpx>=0.25.0", "weaviate-client>=4.16.0,!=4.16.7", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.1.2,<3; python_version <"3.13"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"', - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - 'pandas>=2.3.3,<3; python_version >="3.14"', + 'pandas>=2.1.2; python_version <"3.13"', + 'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"', + 'pandas>=2.3.3; python_version >="3.14"', ] [dependency-groups] diff --git a/providers/yandex/docs/changelog.rst b/providers/yandex/docs/changelog.rst index f3673bb39b6cd..d53e266882b59 100644 --- a/providers/yandex/docs/changelog.rst +++ b/providers/yandex/docs/changelog.rst @@ -34,6 +34,7 @@ Bug Fixes ~~~~~~~~~ * ``Refuse the team agnostic fall-through for a team scoped Lockbox secret name (#70877)`` +* ``Only refuse team scoped like secret ids when multi_team is on (#71078)`` .. Below changes are excluded from the changelog. Move them to appropriate section above if needed. Do not delete the lines(!): diff --git a/providers/yandex/src/airflow/providers/yandex/secrets/lockbox.py b/providers/yandex/src/airflow/providers/yandex/secrets/lockbox.py index 4cf8f8c2c7762..e102044a98043 100644 --- a/providers/yandex/src/airflow/providers/yandex/secrets/lockbox.py +++ b/providers/yandex/src/airflow/providers/yandex/secrets/lockbox.py @@ -30,6 +30,7 @@ import yandexcloud from airflow.models import Connection +from airflow.providers.common.compat.sdk import conf from airflow.providers.yandex.utils.credentials import get_credentials from airflow.providers.yandex.utils.defaults import default_conn_name from airflow.providers.yandex.utils.fields import get_field_from_extras @@ -272,7 +273,12 @@ def _names_a_team_namespace(self, secret_id: str) -> bool: the caller's own team builds looks equivalent and is not -- a caller in team ``a`` would match ``a//b``'s namespace on the prefix and read its secrets. Only the caller's own namespace is ever constructed, never parsed. + + Only checked in multi-team mode: ``team_name`` is never non-``None`` otherwise, so no + team scoped secret can exist to collide with. """ + if not conf.getboolean("core", "multi_team", fallback=False): + return False return self.sep * TEAM_SEP_MULTIPLIER in secret_id def _log_refusal(self, kind: str, secret_id: str) -> None: diff --git a/providers/yandex/tests/unit/yandex/secrets/test_lockbox.py b/providers/yandex/tests/unit/yandex/secrets/test_lockbox.py index d50a9f9b947db..311e364105e99 100644 --- a/providers/yandex/tests/unit/yandex/secrets/test_lockbox.py +++ b/providers/yandex/tests/unit/yandex/secrets/test_lockbox.py @@ -30,6 +30,10 @@ from airflow.providers.yandex.secrets.lockbox import LockboxSecretBackend from airflow.providers.yandex.utils.defaults import default_conn_name +from tests_common.test_utils.config import conf_vars + +multi_team_enabled = conf_vars({("core", "multi_team"): "True"}) + class TestLockboxSecretBackend: @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secret_value") @@ -305,6 +309,7 @@ def test_get_variable_falls_back_to_global_secret_when_team_secret_is_missing( assert result == "global-value" mock_get_payload.assert_called_once_with("456", ANY) + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") def test_get_variable_returns_none_for_team_scoped_key_without_team_name(self, mock_get_secrets): backend = LockboxSecretBackend() @@ -312,6 +317,7 @@ def test_get_variable_returns_none_for_team_scoped_key_without_team_name(self, m assert backend.get_variable("teama//hello") is None mock_get_secrets.assert_not_called() + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") def test_get_conn_value_returns_none_for_team_scoped_id_without_team_name(self, mock_get_secrets): backend = LockboxSecretBackend() @@ -319,6 +325,7 @@ def test_get_conn_value_returns_none_for_team_scoped_id_without_team_name(self, assert backend.get_conn_value("teama//my_db") is None mock_get_secrets.assert_not_called() + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_payload") def test_another_teams_secret_is_not_reachable(self, mock_get_payload, mock_get_secrets): @@ -339,6 +346,7 @@ def test_another_teams_secret_is_not_reachable(self, mock_get_payload, mock_get_ assert backend.get_conn_value("teama//my_db", team_name="teamb") is None mock_get_payload.assert_not_called() + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_payload") def test_team_whose_name_extends_the_callers_is_not_reachable(self, mock_get_payload, mock_get_secrets): @@ -355,6 +363,7 @@ def test_team_whose_name_extends_the_callers_is_not_reachable(self, mock_get_pay assert backend.get_conn_value("teama//prod//my_db", team_name="teama") is None mock_get_payload.assert_not_called() + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_payload") def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace( @@ -379,6 +388,7 @@ def test_team_scoped_lookup_cannot_reach_a_longer_teams_namespace( assert backend.get_conn_value("prod//my_db", team_name="teama") is None mock_get_payload.assert_not_called() + @multi_team_enabled @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") def test_refusing_an_ambiguous_id_is_logged(self, mock_get_secrets, caplog): """A silent ``None`` is indistinguishable from a missing secret, so the refusal is logged.""" @@ -392,6 +402,22 @@ def test_refusing_an_ambiguous_id_is_logged(self, mock_get_secrets, caplog): assert all(r.levelname == "WARNING" for r in refusals) mock_get_secrets.assert_not_called() + @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") + @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_payload") + def test_ambiguous_id_resolves_when_multi_team_is_disabled(self, mock_get_payload, mock_get_secrets): + """No team scoped secret can exist without multi-team mode, so there is no ambiguity + to refuse -- an ordinary id containing the separator must resolve normally.""" + mock_get_secrets.return_value = [ + secret_pb.Secret(id="123", name="airflow/connections/prod//my_db"), + ] + mock_get_payload.return_value = payload_pb.Payload( + entries=[payload_pb.Payload.Entry(text_value="prod-conn")] + ) + + backend = LockboxSecretBackend() + + assert backend.get_conn_value("prod//my_db") == "prod-conn" + @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_secrets") @patch("airflow.providers.yandex.secrets.lockbox.LockboxSecretBackend._get_payload") def test_yandex_lockbox_secret_backend__get_secret_value(self, mock_get_payload, mock_get_secrets): diff --git a/pyproject.toml b/pyproject.toml index be805a23ca0ce..b09bdf933d02b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -191,9 +191,6 @@ apache-airflow = "airflow.__main__:main" "common.compat" = [ "apache-airflow-providers-common-compat>=1.2.1" ] -"common.dataquality" = [ - "apache-airflow-providers-common-dataquality>=0.1.0" -] "common.io" = [ "apache-airflow-providers-common-io>=1.4.2" ] @@ -257,9 +254,6 @@ apache-airflow = "airflow.__main__:main" "http" = [ "apache-airflow-providers-http>=4.13.2" ] -"ibm.mq" = [ - "apache-airflow-providers-ibm-mq>=0.1.0; platform_machine !=\"aarch64\" and platform_machine !=\"arm64\"" -] "imap" = [ "apache-airflow-providers-imap>=3.8.0" ] @@ -444,7 +438,6 @@ apache-airflow = "airflow.__main__:main" "apache-airflow-providers-cohere>=1.4.0", "apache-airflow-providers-common-ai>=0.1.0", "apache-airflow-providers-common-compat>=1.2.1", - "apache-airflow-providers-common-dataquality>=0.1.0", "apache-airflow-providers-common-io>=1.4.2", "apache-airflow-providers-common-messaging>=2.0.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py "apache-airflow-providers-common-sql>=1.18.0", @@ -466,7 +459,6 @@ apache-airflow = "airflow.__main__:main" "apache-airflow-providers-grpc>=3.7.0", "apache-airflow-providers-hashicorp>=4.0.0", "apache-airflow-providers-http>=4.13.2", - "apache-airflow-providers-ibm-mq>=0.1.0; platform_machine !=\"aarch64\" and platform_machine !=\"arm64\"", "apache-airflow-providers-imap>=3.8.0", "apache-airflow-providers-influxdb>=2.8.0", "apache-airflow-providers-informatica>=0.1.1", @@ -609,6 +601,10 @@ extend-exclude = [ [tool.ruff.lint] typing-modules = ["airflow.typing_compat"] external = ["MIG"] +# Pin the base rule set instead of inheriting Ruff's default, which Ruff changes +# between releases (0.16 widened it from ~250 to ~510 rules). Everything Airflow +# actually opts into lives in extend-select below. +select = ["E4", "E7", "E9", "F"] extend-select = [ # Enable entire ruff rule section "I", # Missing required import (auto-fixable) @@ -716,6 +712,13 @@ ignore = [ "COM812", "COM819", "E501", # Formatted code may exceed the line length, leading to line-too-long (E501) errors. + # New in Ruff 0.16, inside sections Airflow selects wholesale. Deferred so a tooling + # bump does not also carry semantic changes; both need per-site judgement. + "ISC004", # Implicit string concat in a collection literal; 109 sites, unsafe autofix + # LOG004 is not a mechanical replacement: switching to `.error()` where an exception is + # in flight silently drops the traceback, while leaving `.exception()` where none is + # only prints a bare `NoneType: None`. Review already caught one bad guess in 13. + "LOG004", # `.exception()` outside an exception handler; 13 sites ] unfixable = [ # PT022 replace empty `yield` to empty `return`. Might be fixed with a combination of PLR1711 diff --git a/registry/.eleventy.js b/registry/.eleventy.js index bdd5ce7b20c55..6a61ca7d792b4 100644 --- a/registry/.eleventy.js +++ b/registry/.eleventy.js @@ -99,6 +99,28 @@ module.exports = function(eleventyConfig) { return JSON.stringify(obj); }); + // Attaches each type's module count and drops zero-count types, sorted by + // count descending -- used to render the provider-page module tabs with the + // busiest categories first. + eleventyConfig.addFilter("sortTypesByCount", (types, moduleCounts) => { + if (!Array.isArray(types)) return []; + const counts = moduleCounts || {}; + return types + .filter((t) => (counts[t.id] || 0) > 0) + .map((t) => Object.assign({}, t, { count: counts[t.id] || 0 })) + .sort((a, b) => b.count - a.count); + }); + + // Looks up the curated single-letter icon for a type id from types.json, instead of + // deriving it from the id's first character at each render site (which reintroduces + // the letter collisions types.json's `icon` field was curated to avoid). + eleventyConfig.addFilter("typeIcon", (types, typeId) => { + if (!Array.isArray(types) || !typeId) return ""; + const match = types.find((t) => t.id === typeId); + if (match && match.icon) return match.icon; + return typeId.charAt(0).toUpperCase(); + }); + eleventyConfig.addShortcode("year", () => `${new Date().getFullYear()}`); return { diff --git a/registry/src/_data/types.json b/registry/src/_data/types.json index 150f986b2e4c5..cb4d36f0166dc 100644 --- a/registry/src/_data/types.json +++ b/registry/src/_data/types.json @@ -63,5 +63,35 @@ "id": "retry_policy", "label": "Retry Policies", "icon": "R" + }, + { + "id": "extra_link", + "label": "Extra Links", + "icon": "I" + }, + { + "id": "queue", + "label": "Message Queues", + "icon": "Q" + }, + { + "id": "plugin", + "label": "Plugins", + "icon": "P" + }, + { + "id": "auth_manager", + "label": "Auth Managers", + "icon": "A" + }, + { + "id": "db_manager", + "label": "DB Managers", + "icon": "M" + }, + { + "id": "dialect", + "label": "Dialects", + "icon": "D" } ] diff --git a/registry/src/css/main.css b/registry/src/css/main.css index 8a262428b55b5..ef006000248c1 100644 --- a/registry/src/css/main.css +++ b/registry/src/css/main.css @@ -898,6 +898,14 @@ footer a:hover { .provider-card .modules .logging { background: var(--color-logging); } .provider-card .modules .bundle { background: var(--color-bundle); } .provider-card .modules .decorator { background: var(--color-decorator); } +.provider-card .modules .toolset { background: var(--color-toolset); } +.provider-card .modules .retry_policy { background: var(--color-retry_policy); } +.provider-card .modules .extra_link { background: var(--color-extra_link); } +.provider-card .modules .queue { background: var(--color-queue); } +.provider-card .modules .plugin { background: var(--color-plugin); } +.provider-card .modules .auth_manager { background: var(--color-auth_manager); } +.provider-card .modules .db_manager { background: var(--color-db_manager); } +.provider-card .modules .dialect { background: var(--color-dialect); } .provider-card .meta, #featured-providers .meta { @@ -2158,6 +2166,45 @@ main { color: var(--color-decorator); } +.type-icon.toolset { + background: rgb(from var(--color-toolset) r g b / 0.2); + color: var(--color-toolset); +} + +.type-icon.retry_policy { + background: rgb(from var(--color-retry_policy) r g b / 0.2); + color: var(--color-retry_policy); +} + +.type-icon.extra_link { + background: rgb(from var(--color-extra_link) r g b / 0.2); + color: var(--color-extra_link); +} + +.type-icon.queue { + background: rgb(from var(--color-queue) r g b / 0.2); + color: var(--color-queue); +} + +.type-icon.plugin { + background: rgb(from var(--color-plugin) r g b / 0.2); + color: var(--color-plugin); +} + +.type-icon.auth_manager { + background: rgb(from var(--color-auth_manager) r g b / 0.2); + color: var(--color-auth_manager); +} + +.type-icon.db_manager { + background: rgb(from var(--color-db_manager) r g b / 0.2); + color: var(--color-db_manager); +} + +.type-icon.dialect { + background: rgb(from var(--color-dialect) r g b / 0.2); + color: var(--color-dialect); +} .type-count { font-size: var(--text-2xl); @@ -2246,6 +2293,37 @@ main { --meter-color: var(--color-decorator); } +.share-bar.toolset { + --meter-color: var(--color-toolset); +} + +.share-bar.retry_policy { + --meter-color: var(--color-retry_policy); +} + +.share-bar.extra_link { + --meter-color: var(--color-extra_link); +} + +.share-bar.queue { + --meter-color: var(--color-queue); +} + +.share-bar.plugin { + --meter-color: var(--color-plugin); +} + +.share-bar.auth_manager { + --meter-color: var(--color-auth_manager); +} + +.share-bar.db_manager { + --meter-color: var(--color-db_manager); +} + +.share-bar.dialect { + --meter-color: var(--color-dialect); +} .share-label { font-size: var(--text-xs); @@ -3067,6 +3145,55 @@ main { .tab-icon.decorator { background: rgb(from var(--color-decorator) r g b / 0.2); color: var(--color-decorator); } .tab-icon.toolset { background: rgb(from var(--color-toolset) r g b / 0.2); color: var(--color-toolset); } .tab-icon.retry_policy { background: rgb(from var(--color-retry_policy) r g b / 0.2); color: var(--color-retry_policy); } +.tab-icon.extra_link { background: rgb(from var(--color-extra_link) r g b / 0.2); color: var(--color-extra_link); } +.tab-icon.queue { background: rgb(from var(--color-queue) r g b / 0.2); color: var(--color-queue); } +.tab-icon.plugin { background: rgb(from var(--color-plugin) r g b / 0.2); color: var(--color-plugin); } +.tab-icon.auth_manager { background: rgb(from var(--color-auth_manager) r g b / 0.2); color: var(--color-auth_manager); } +.tab-icon.db_manager { background: rgb(from var(--color-db_manager) r g b / 0.2); color: var(--color-db_manager); } +.tab-icon.dialect { background: rgb(from var(--color-dialect) r g b / 0.2); color: var(--color-dialect); } + +/* Module Tabs "More" overflow menu */ +.module-tab-more { + position: relative; +} + +.module-tab-more-btn { + display: inline-flex; +} + +.module-tab-more-btn svg { + width: 1rem; + height: 1rem; + transition: transform var(--transition-base); +} + +.module-tab-more-btn[aria-expanded="true"] svg { + transform: rotate(180deg); +} + +.module-tab-more-menu { + position: absolute; + top: calc(100% + var(--space-2)); + left: 0; + z-index: 20; + display: flex; + flex-direction: column; + gap: var(--space-1); + min-width: 12rem; + padding: var(--space-2); + background: var(--bg-secondary); + border: 1px solid var(--border-primary); + border-radius: var(--radius-lg); + box-shadow: var(--shadow-lg); +} + +.module-tab-more-menu[hidden] { + display: none; +} + +.module-tab-more-menu .module-tab { + justify-content: flex-start; +} /* Modules Layout (sidebar + content) */ .modules-layout { @@ -3242,6 +3369,84 @@ main { border: 1px solid rgb(from var(--color-bundle) r g b / 0.3); } +.provider-detail-page .modules .module .icon.notifier { + background: rgb(from var(--color-notifier) r g b / 0.2); + color: var(--color-notifier); + border: 1px solid rgb(from var(--color-notifier) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.secret { + background: rgb(from var(--color-secret) r g b / 0.2); + color: var(--color-secret); + border: 1px solid rgb(from var(--color-secret) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.logging { + background: rgb(from var(--color-logging) r g b / 0.2); + color: var(--color-logging); + border: 1px solid rgb(from var(--color-logging) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.executor { + background: rgb(from var(--color-executor) r g b / 0.2); + color: var(--color-executor); + border: 1px solid rgb(from var(--color-executor) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.decorator { + background: rgb(from var(--color-decorator) r g b / 0.2); + color: var(--color-decorator); + border: 1px solid rgb(from var(--color-decorator) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.toolset { + background: rgb(from var(--color-toolset) r g b / 0.2); + color: var(--color-toolset); + border: 1px solid rgb(from var(--color-toolset) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.retry_policy { + background: rgb(from var(--color-retry_policy) r g b / 0.2); + color: var(--color-retry_policy); + border: 1px solid rgb(from var(--color-retry_policy) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.extra_link { + background: rgb(from var(--color-extra_link) r g b / 0.2); + color: var(--color-extra_link); + border: 1px solid rgb(from var(--color-extra_link) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.queue { + background: rgb(from var(--color-queue) r g b / 0.2); + color: var(--color-queue); + border: 1px solid rgb(from var(--color-queue) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.plugin { + background: rgb(from var(--color-plugin) r g b / 0.2); + color: var(--color-plugin); + border: 1px solid rgb(from var(--color-plugin) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.auth_manager { + background: rgb(from var(--color-auth_manager) r g b / 0.2); + color: var(--color-auth_manager); + border: 1px solid rgb(from var(--color-auth_manager) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.db_manager { + background: rgb(from var(--color-db_manager) r g b / 0.2); + color: var(--color-db_manager); + border: 1px solid rgb(from var(--color-db_manager) r g b / 0.3); +} + +.provider-detail-page .modules .module .icon.dialect { + background: rgb(from var(--color-dialect) r g b / 0.2); + color: var(--color-dialect); + border: 1px solid rgb(from var(--color-dialect) r g b / 0.3); +} + .provider-detail-page .modules .module .content { flex: 1; min-width: 0; @@ -3556,6 +3761,40 @@ main { border-color: var(--color-green-400); } +/* Per-connection external services matrix */ +.provider-detail-page .connections-card .conn-external-services-table { + width: 100%; + margin-top: var(--space-4); + border-collapse: collapse; +} + +.provider-detail-page .connections-card .conn-external-services-header th { + padding: var(--space-2) var(--space-3); + font-size: var(--text-sm); + font-weight: var(--font-semibold); + color: var(--text-secondary); + text-align: left; + background: var(--bg-tertiary); +} + +.provider-detail-page .connections-card .conn-external-services-row { + border-top: 1px solid var(--border-primary); +} + +.provider-detail-page .connections-card .conn-external-services-row td { + padding: var(--space-2) var(--space-3); + vertical-align: top; +} + +.provider-detail-page .connections-card .conn-external-services-row td:first-child { + white-space: nowrap; +} + +.provider-detail-page .connections-card .conn-external-services-list { + color: var(--text-secondary); + font-size: var(--text-sm); +} + /* Shared builder panel */ .provider-detail-page .connections-card .conn-builder-panel { margin-top: var(--space-3); diff --git a/registry/src/css/tokens.css b/registry/src/css/tokens.css index 74a6c039e006f..5d8b27897f8a4 100644 --- a/registry/src/css/tokens.css +++ b/registry/src/css/tokens.css @@ -67,6 +67,12 @@ --color-decorator: #d946ef; --color-toolset: #84cc16; --color-retry_policy: #10b981; + --color-extra_link: #06b6d4; + --color-queue: #eab308; + --color-plugin: #8b5cf6; + --color-auth_manager: #ef4444; + --color-db_manager: #64748b; + --color-dialect: #f472b6; /* Additional Colors */ --color-green-400: #4ade80; diff --git a/registry/src/js/provider-detail.js b/registry/src/js/provider-detail.js index df6a81c02e145..2ecb7a52f9902 100644 --- a/registry/src/js/provider-detail.js +++ b/registry/src/js/provider-detail.js @@ -25,7 +25,9 @@ var extraDepsList = document.getElementById('extra-deps-list'); var extrasDataEl = document.getElementById('extras-data'); var moduleSearch = document.getElementById('module-search'); - var moduleTabs = document.querySelectorAll('.module-tab'); + // [data-type] excludes the "More" toggle button, which is a .module-tab + // for styling purposes only and has no data-type of its own. + var moduleTabs = document.querySelectorAll('.module-tab[data-type]'); var categoryBtns = document.querySelectorAll('.category-btn'); var moduleItems = document.querySelectorAll('.modules .module'); var copyImportBtns = document.querySelectorAll('.copy-import'); @@ -152,9 +154,52 @@ tab.classList.add('active'); currentType = tab.dataset.type || 'all'; filterModules(); + if (moduleTabMoreBtn && moduleTabMoreMenu) { + moduleTabMoreBtn.classList.toggle('active', moduleTabMoreMenu.contains(tab)); + } }); }); + // "More" overflow menu for module tabs that don't fit in the visible row. + var moduleTabMoreBtn = document.getElementById('module-tab-more-btn'); + var moduleTabMoreMenu = document.getElementById('module-tab-more-menu'); + + if (moduleTabMoreBtn && moduleTabMoreMenu) { + function closeMoreMenu() { + var focusWasInMenu = moduleTabMoreMenu.contains(document.activeElement); + moduleTabMoreBtn.setAttribute('aria-expanded', 'false'); + moduleTabMoreMenu.hidden = true; + if (focusWasInMenu) { moduleTabMoreBtn.focus(); } + } + + moduleTabMoreBtn.addEventListener('click', function(e) { + e.stopPropagation(); + var isOpen = moduleTabMoreBtn.getAttribute('aria-expanded') === 'true'; + if (isOpen) { + closeMoreMenu(); + } else { + moduleTabMoreBtn.setAttribute('aria-expanded', 'true'); + moduleTabMoreMenu.hidden = false; + } + }); + + document.addEventListener('click', function(e) { + if (!moduleTabMoreMenu.contains(e.target) && e.target !== moduleTabMoreBtn) { + closeMoreMenu(); + } + }); + + document.addEventListener('keydown', function(e) { + if (e.key === 'Escape') closeMoreMenu(); + }); + + // Selecting a tab inside the menu also closes it (filtering itself is + // handled by the shared moduleTabs click listener above). + moduleTabMoreMenu.querySelectorAll('.module-tab').forEach(function(tab) { + tab.addEventListener('click', closeMoreMenu); + }); + } + categoryBtns.forEach(function(btn) { btn.addEventListener('click', function() { categoryBtns.forEach(function(b) { b.classList.remove('active'); }); diff --git a/registry/src/js/search.js b/registry/src/js/search.js index d7b13d523d636..1f984183418e5 100644 --- a/registry/src/js/search.js +++ b/registry/src/js/search.js @@ -24,17 +24,19 @@ let currentResults = []; let searchId = 0; - // Type labels loaded from types.json (injected via base.njk) + // Type labels/icons loaded from types.json (injected via base.njk) const typeLabels = {}; + const typeIcons = {}; try { const typesEl = document.getElementById('types-data'); if (typesEl) { for (const t of JSON.parse(typesEl.textContent)) { typeLabels[t.id] = t.label; + typeIcons[t.id] = t.icon; } } } catch (_) { - // Fallback: empty object — badges will show raw type name + // Fallback: empty objects — badges show raw type name, icons fall back to first letter } function escapeHtml(str) { @@ -105,7 +107,7 @@ const providerName = result.meta.providerName || ''; const description = result.meta.description || result.excerpt; const moduleType = result.meta.moduleType || ''; - const icon = type === 'provider' ? 'P' : (moduleType ? moduleType[0].toUpperCase() : 'M'); + const icon = type === 'provider' ? 'P' : (moduleType ? (typeIcons[moduleType] || moduleType[0].toUpperCase()) : 'M'); const resultType = type === 'provider' ? 'provider' : moduleType; return ` diff --git a/registry/src/provider-version.njk b/registry/src/provider-version.njk index 49356c0796fc6..ec56d1167f84d 100644 --- a/registry/src/provider-version.njk +++ b/registry/src/provider-version.njk @@ -248,6 +248,30 @@ eleventyComputed: {% endfor %} + {% set hasExternalServices = false %} + {% for conn in conns %} + {% if conn.external_services and conn.external_services.length > 0 %}{% set hasExternalServices = true %}{% endif %} + {% endfor %} + {% if hasExternalServices %} + + + + + + + + + {% for conn in conns %} + {% if conn.external_services and conn.external_services.length > 0 %} + + + + + {% endif %} + {% endfor %} + +
Connection typeExternal services (examples)
{{ conn.conn_type or conn.connection_type }}{{ conn.external_services | join(", ") }}
+ {% endif %}
{% if totalModules > 0 %} - {# Module Type Tabs #} + {# Module Type Tabs: sorted by module count (busiest first), overflow into a "More" menu #} + {% set moduleTabsVisibleCount = 8 %} + {% set sortedTypes = types | sortTypesByCount(moduleCounts) %} + {% set visibleTypes = sortedTypes | slice(0, moduleTabsVisibleCount) %} + {% set overflowTypes = sortedTypes | slice(moduleTabsVisibleCount, sortedTypes.length) %} {# Main content with sidebar #} @@ -352,7 +395,7 @@ eleventyComputed: {% for module in providerModules %}
- {{ module.type[0] | upper }} + {{ types | typeIcon(module.type) }}

diff --git a/scripts/ci/notify_uv_lock_conflicts.py b/scripts/ci/notify_uv_lock_conflicts.py index d22614b079a8b..a78edb3d161bf 100644 --- a/scripts/ci/notify_uv_lock_conflicts.py +++ b/scripts/ci/notify_uv_lock_conflicts.py @@ -237,6 +237,15 @@ def resolve_source_pr( return None +def avoid_backlink(url: str) -> str: + """Swap ``github.com`` for ``redirect.github.com`` so linking to the source + PR/commit doesn't generate an automatic backlink comment on it. + + See: https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls#avoiding-backlinks-to-linked-references + """ + return url.replace("https://github.com/", "https://redirect.github.com/", 1) + + def build_body(source_ref_md: str) -> str: return "\n".join( [ @@ -424,13 +433,16 @@ def main() -> int: with GitHubGraphQL(token) as client: source_pr = resolve_source_pr(client, owner, repo, sha, short_sha) if source_pr: + # Use redirect.github.com for both links: they point at the PR/commit + # that caused this notice, and a plain github.com link would generate + # an unwanted backlink comment there. source_ref_md = ( - f"[#{source_pr['number']}]({source_pr['url']}) " - f'("{source_pr["title"]}"), commit [`{short_sha}`]({commit_url})' + f"[#{source_pr['number']}]({avoid_backlink(source_pr['url'])}) " + f'("{source_pr["title"]}"), commit [`{short_sha}`]({avoid_backlink(commit_url)})' ) source_ref_plain = f"#{source_pr['number']} ({source_pr['url']}) — commit {short_sha}" else: - source_ref_md = f"commit [`{short_sha}`]({commit_url})" + source_ref_md = f"commit [`{short_sha}`]({avoid_backlink(commit_url)})" source_ref_plain = f"commit {short_sha}" log(f"Source of uv.lock change: {source_ref_plain}") diff --git a/scripts/ci/prek/check_airflowctl_command_coverage.py b/scripts/ci/prek/check_airflowctl_command_coverage.py index 44ef0c34bb8e3..a8a767453fcba 100755 --- a/scripts/ci/prek/check_airflowctl_command_coverage.py +++ b/scripts/ci/prek/check_airflowctl_command_coverage.py @@ -53,7 +53,7 @@ "assets delete-dag-queued-events", "assets delete-queued-event", "assets delete-queued-events", - "assets get-by-alias", + "assets get-alias", "assets get-dag-queued-event", "assets get-dag-queued-events", "assets get-queued-events", diff --git a/scripts/ci/prek/check_imports_in_providers.py b/scripts/ci/prek/check_imports_in_providers.py index f67b57956f680..f6710c6b219b9 100755 --- a/scripts/ci/prek/check_imports_in_providers.py +++ b/scripts/ci/prek/check_imports_in_providers.py @@ -19,7 +19,7 @@ # requires-python = ">=3.10,<3.11" # dependencies = [ # "rich>=13.6.0", -# "ruff==0.15.22", +# "ruff==0.16.0", # ] # /// from __future__ import annotations diff --git a/scripts/ci/prek/common_prek_utils.py b/scripts/ci/prek/common_prek_utils.py index e316eb154aea6..559c2f0a047ae 100644 --- a/scripts/ci/prek/common_prek_utils.py +++ b/scripts/ci/prek/common_prek_utils.py @@ -591,19 +591,30 @@ def get_provider_base_dir_from_path(file_path: Path) -> Path | None: return None -def get_all_provider_ids(exclude_suspended_providers: bool = False) -> list[str]: +def get_all_provider_ids( + exclude_suspended_providers: bool = False, exclude_not_ready_providers: bool = False +) -> list[str]: """ Get all providers from the new provider structure + + :param exclude_suspended_providers: skip providers whose state is ``suspended`` + :param exclude_not_ready_providers: skip providers whose state is ``not-ready`` - those have + never been published, so anything describing what is installable must leave them out """ all_provider_ids = [] + excluded_states = set() + if exclude_suspended_providers: + excluded_states.add("suspended") + if exclude_not_ready_providers: + excluded_states.add("not-ready") for provider_file in AIRFLOW_PROVIDERS_ROOT_PATH.rglob("provider.yaml"): if provider_file.is_relative_to(AIRFLOW_PROVIDERS_ROOT_PATH / "src"): continue - if exclude_suspended_providers: + if excluded_states: import yaml provider_info = yaml.safe_load(provider_file.read_text()) - if provider_info.get("state") == "suspended": + if provider_info.get("state") in excluded_states: continue provider_id = get_provider_id_from_path(provider_file) if provider_id: diff --git a/scripts/ci/prek/ruff_format.py b/scripts/ci/prek/ruff_format.py index 8c82e3e48df54..df7a84cfeb0ab 100755 --- a/scripts/ci/prek/ruff_format.py +++ b/scripts/ci/prek/ruff_format.py @@ -18,7 +18,7 @@ # /// script # requires-python = ">=3.10,<3.11" # dependencies = [ -# "ruff==0.15.22", +# "ruff==0.16.0", # ] # /// diff --git a/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py b/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py index 66b8db86f6ea8..3bcd31c6886b5 100755 --- a/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py +++ b/scripts/ci/prek/run_mypy_full_dist_local_venv_or_breeze_in_ci.py @@ -303,8 +303,8 @@ def run_local_mypy(project: str, hook_name: str, files: list[str], config_file: print("No files to test. Quitting") sys.exit(0) - write_file_list(all_files_to_check) - file_argument_ci = "@/files/mypy_files.txt" + mypy_file_list = write_file_list(all_files_to_check, suffix=mypy_folders[0].replace("/", "-")) + file_argument_ci = f"@/files/{mypy_file_list.name}" mypy_cmd = ( f"TERM=ansi mypy --follow-imports=silent {shlex.quote(file_argument_ci)} {' '.join(mypy_extra_args)}" diff --git a/scripts/ci/prek/update_airflow_pyproject_toml.py b/scripts/ci/prek/update_airflow_pyproject_toml.py index 2313790e0a5d5..f66a3d7ba5d49 100755 --- a/scripts/ci/prek/update_airflow_pyproject_toml.py +++ b/scripts/ci/prek/update_airflow_pyproject_toml.py @@ -269,9 +269,20 @@ def get_exclusion_marker(provider_dependencies: dict[str, Any]) -> str: all_optional_dependencies.append(f'"{optional}" = [\n "apache-airflow-core[{optional}]"\n]\n') optional_airflow_task_sdk_dependencies = get_optional_dependencies(AIRFLOW_TASK_SDK_PYPROJECT_TOML_FILE) all_optional_dependencies.append('"all-task-sdk" = [\n "apache-airflow-task-sdk[all]"\n]\n') + # Two lists, because the sections below describe two different things. + # + # `all_providers` describes the source tree: mypy has to type-check a not-ready provider and uv + # has to keep it in the workspace, which is how CI installs it. Only suspended providers drop out. + # + # `released_providers` describes what is installable from PyPI. A not-ready provider has never + # been published, so naming it in an extra makes that extra unsatisfiable - there is no version + # of it to resolve to. all_providers = sorted(get_all_provider_ids(exclude_suspended_providers=True)) + released_providers = sorted( + get_all_provider_ids(exclude_suspended_providers=True, exclude_not_ready_providers=True) + ) all_provider_lines = [] - for provider_id in all_providers: + for provider_id in released_providers: distribution_name = provider_distribution_name(provider_id) min_provider_version, comment = find_min_provider_version(provider_id) exclusion_marker = get_exclusion_marker(all_providers_dependencies.get(provider_id, {})) @@ -288,10 +299,15 @@ def get_exclusion_marker(provider_dependencies: dict[str, Any]) -> str: all_provider_lines.append(f' "{distribution_name}",\n') all_optional_dependencies.append('"all" = [\n') optional_apache_airflow_dependencies = get_optional_dependencies(AIRFLOW_PYPROJECT_TOML_FILE) + # Filtered against every provider id rather than against `all_providers`: a provider left out of + # `all_providers` still has an extra named after it, and testing only against the included ones + # would sweep that extra in here instead of dropping it - which is how a not-ready provider would + # come back into `all` through the side door. + every_provider_id = set(get_all_provider_ids()) all_local_extras = [ extra for extra in sorted(optional_apache_airflow_dependencies) - if extra not in all_providers and not extra.startswith("all") + if extra not in every_provider_id and not extra.startswith("all") ] all_optional_dependencies.append(f' "apache-airflow[{",".join(all_local_extras)}]",\n') all_optional_dependencies.append(' "apache-airflow-core[all]",\n') diff --git a/scripts/ci/prek/validate_operators_init.py b/scripts/ci/prek/validate_operators_init.py index 664d3e4e471e7..86d25180508ad 100755 --- a/scripts/ci/prek/validate_operators_init.py +++ b/scripts/ci/prek/validate_operators_init.py @@ -403,9 +403,11 @@ def _check_constructor_field_logic( if findings: console.print( f"{class_node.name}'s constructor applies logic to template fields. Template fields " - f"are rendered after the constructor runs, so validation or transformation here acts " - f"on the un-rendered Jinja expression and should move to execute() " - f"(see contributing-docs/05_pull_requests.rst):" + f"are rendered after the constructor runs, so value-dependent validation or transformation " + f"here acts on the un-rendered Jinja expression and should run in execute() or another method " + f"called after rendering. Checks that only ask whether an argument was passed belong in " + f"__init__ and should use explicit is None / is not None comparisons rather than truthiness. " + f"See https://github.com/apache/airflow/issues/70296#false-positives" ) for lineno in sorted(findings): source = source_lines[lineno - 1].strip() if lineno <= len(source_lines) else "" diff --git a/scripts/in_container/run_generate_constraints.py b/scripts/in_container/run_generate_constraints.py index 6310ec1a05211..6bd75777253f4 100755 --- a/scripts/in_container/run_generate_constraints.py +++ b/scripts/in_container/run_generate_constraints.py @@ -21,15 +21,19 @@ import json import os import sys +from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass from datetime import datetime from functools import cached_property +from itertools import repeat from pathlib import Path from typing import TextIO import requests from click import Choice from in_container_utils import AIRFLOW_DIST_PATH, AIRFLOW_ROOT_PATH, click, console, run_command +from packaging.specifiers import InvalidSpecifier, SpecifierSet +from packaging.version import InvalidVersion, Version try: import tomllib @@ -39,6 +43,8 @@ DEFAULT_BRANCH = os.environ.get("DEFAULT_BRANCH", "main") PYTHON_VERSION = os.environ.get("PYTHON_MAJOR_MINOR_VERSION", "3.10") GENERATED_PROVIDER_DEPENDENCIES_FILE = AIRFLOW_ROOT_PATH / "generated" / "provider_dependencies.json" +PYPI_JSON_API_URL = "https://pypi.org/pypi/{distribution}/json" +PYPI_LOOKUP_PARALLELISM = 8 def _read_version_from_pyproject(pyproject_path: Path) -> str: @@ -130,6 +136,7 @@ def _read_dynamic_version_from_init(init_path: Path) -> str: @dataclass class ConfigParams: airflow_constraints_mode: str + allow_pre_releases: bool constraints_github_repository: str default_constraints_branch: str github_actions: bool @@ -379,11 +386,19 @@ def check_providers_not_downgraded(config_params: ConfigParams) -> None: Fail generation if any released provider is downgraded compared to the latest constraints. Released provider versions only ever move forward on PyPI, so a lower version in the freshly - generated constraints signals a resolution problem (a broken dependency forcing an old provider - back in) rather than an intended change. We stop here so it is caught instead of being published. - """ - from packaging.version import InvalidVersion, Version + generated constraints means the version we had pinned is no longer installable - yanked, or + left without a file for the Python being resolved. We stop here so it is caught instead of + being published. + A release candidate is exempt: it pins the wave being voted on, which sorts below the released + versions a constraints branch may already carry, so the comparison says nothing there. + """ + if config_params.allow_pre_releases: + console.print( + "[yellow]Pre-releases are allowed - skipping the provider downgrade check, a candidate " + "sorts below the releases the constraints branch carries." + ) + return if not config_params.latest_constraints_file.exists(): console.print("[yellow]No previous constraints file downloaded - skipping provider downgrade check.") return @@ -407,10 +422,10 @@ def check_providers_not_downgraded(config_params: ConfigParams) -> None: for provider, latest_version, current_version in sorted(downgraded): console.print(f"[red] * {provider}: {latest_version} -> {current_version}") console.print( - "[yellow]Released providers should never be downgraded. This usually means a broken " - "dependency version forced an older provider back in during resolution. Investigate the " - "diff above and, if needed, add an exclusion in the " - f"`additional_constraints_for_highest_resolution` list in [/] {__file__}" + "[yellow]Released providers should never be downgraded. The providers are pinned at the " + "newest version PyPI serves for this Python, so a lower one means the version we had is " + "gone - yanked, or no longer offering a file this Python can install. Investigate the " + "diff above before publishing.[/]" ) write_provider_downgrade_slack_message(config_params, downgraded) sys.exit(1) @@ -458,6 +473,85 @@ def get_all_active_provider_distributions(python_version: str | None = None) -> ] +def is_file_installable(pypi_file: dict, target_python: Version) -> bool: + """Whether a file PyPI lists for a release can be installed on ``target_python``.""" + if pypi_file.get("yanked"): + return False + requires_python = pypi_file.get("requires_python") + if not requires_python: + return True + try: + return target_python in SpecifierSet(requires_python) + except InvalidSpecifier: + return True + + +def find_newest_version_in_pypi( + distribution: str, python_version: str, allow_pre_releases: bool +) -> str | None: + """Return the newest version of ``distribution`` PyPI can install for ``python_version``. + + A pre-release is only ever considered when ``allow_pre_releases`` is set, and even then it wins + only by sorting above every final release - so a provider with a candidate in the wave being + voted on resolves to that candidate while one without keeps its last release. Versions PyPI can + no longer serve for the Python being resolved - fully yanked, or excluded by ``requires_python`` + - are passed over, because pinning one leaves the resolution nothing to install. ``None`` + (nothing installable at all, e.g. a provider whose first release is still in this wave) leaves + the distribution unpinned rather than pinned to a version that cannot be had. + """ + response = requests.get(PYPI_JSON_API_URL.format(distribution=distribution), timeout=60) + if response.status_code == 404: + console.print(f"[yellow]{distribution} is not published in PyPI - leaving it unpinned.") + return None + response.raise_for_status() + target_python = Version(python_version) + newest_version: Version | None = None + for version, files in response.json().get("releases", {}).items(): + try: + parsed_version = Version(version) + except InvalidVersion: + continue + if parsed_version.is_prerelease and not allow_pre_releases: + continue + if not any(is_file_installable(pypi_file, target_python) for pypi_file in files): + continue + if newest_version is None or parsed_version > newest_version: + newest_version = parsed_version + if newest_version is None: + console.print(f"[yellow]{distribution} has no installable version in PyPI - leaving it unpinned.") + return None + return str(newest_version) + + +def build_pinned_provider_requirements(python_version: str, allow_pre_releases: bool) -> list[str]: + """Exact pins for every active provider, at the newest version PyPI holds for it. + + Naming the versions retrieved from PyPI leaves the providers nothing to resolve: the constraints + pin what is actually published rather than whatever the resolver settles on. It is what makes a + release candidate land on the wave being voted on, whose providers are published only as rc + versions - handing uv a pre-release strategy instead left it free to answer with any version + satisfying the lower bounds. It also keeps pre-releases confined to the providers: no + third-party dependency can answer with one, because no requirement here mentions a pre-release + of anything else. + """ + distributions = get_all_active_provider_distributions(python_version) + console.print(f"[bright_blue]Retrieving the newest PyPI version of {len(distributions)} providers.") + with ThreadPoolExecutor(max_workers=PYPI_LOOKUP_PARALLELISM) as executor: + newest_versions = list( + executor.map( + find_newest_version_in_pypi, + distributions, + repeat(python_version), + repeat(allow_pre_releases), + ) + ) + return [ + f"{distribution}=={version}" + for distribution, version in zip(distributions, newest_versions) + if version is not None + ] + + def generate_constraints_source_providers(config_params: ConfigParams) -> None: """ Generates constraints with provider dependencies used from current sources. This might be different @@ -516,11 +610,37 @@ def generate_constraints_pypi_providers(config_params: ConfigParams) -> None: # resolver might choose to downgrade gremlinpython to an older version, # which in turn might downgrade tinkerpop provider. Having gremlinpython>=3.8.0 ensures # that the resolver will not downgrade the provider. + # * opentelemetry-exporter-prometheus>=0.47b0 — this package only ever publishes beta versions + # (airflow-core requires ``>=0.47b0`` and released constraints already pin a beta, e.g. + # ``==0.65b0``). Marking it as a pre-release here rather than relying on uv's if-necessary + # fallback keeps the choice deliberate and identical across every resolution. + # * opentelemetry-semantic-conventions>=0.48b0 — same story: a beta-only package, pulled in as a + # hard dependency of opentelemetry-sdk (via opentelemetry-exporter-otlp and shared/observability). + # The floor only marks it as a pre-release and must stay at the version paired with our + # ``opentelemetry-*>=1.27.0`` floor — opentelemetry-sdk exact-pins the semantic conventions + # version it ships with, so a higher floor here would conflict with any sdk older than that pairing. + # + # These two are the only pre-releases in the constraints we tag, and removing the need for the + # exception is tracked at https://github.com/apache/airflow/issues/71176 + # additional_constraints_for_highest_resolution: list[str] = [ "pyarrow>=22.0.0; python_version >= '3.14'", "gremlinpython>=3.8.0", + "opentelemetry-exporter-prometheus>=0.47b0", + "opentelemetry-semantic-conventions>=0.48b0", ] + # Every run pins the providers at the versions PyPI holds when it is generated. Only a run for + # a release candidate considers the rc versions of the wave being voted on; a final regenerates + # these against the published releases, so a released constraints file can never carry an rc pin. + pinned_provider_requirements = build_pinned_provider_requirements( + config_params.python, config_params.allow_pre_releases + ) + console.print( + f"[bright_blue]Pinning {len(pinned_provider_requirements)} provider distributions to the " + f"{'newest' if config_params.allow_pre_releases else 'newest final'} versions in PyPI." + ) + result = run_command( cmd=[ "uv", @@ -534,6 +654,7 @@ def generate_constraints_pypi_providers(config_params: ConfigParams) -> None: f"apache-airflow-task-sdk=={AIRFLOW_TASK_SDK_VERSION}", "./airflow-ctl", *additional_constraints_for_highest_resolution, + *pinned_provider_requirements, "--reinstall", # We need to pull the provider distributions from PyPI or dist, not the local ones "--resolution", "highest", @@ -546,7 +667,9 @@ def generate_constraints_pypi_providers(config_params: ConfigParams) -> None: if result.returncode != 0: console.print( "[red]Failed to install airflow with PyPI providers with highest resolution.[/]\n" - "[yellow]Please check the output above for details. One of they ways how to resolve it, in " + "[yellow]Please check the output above for details. The providers are pinned at the newest " + "versions PyPI serves, so two of them requiring incompatible dependencies fails here rather " + "than quietly settling on an older provider. One of they ways how to resolve it, in " "case it is caused by a specific broken dependency version, is to exclude it above in the " f"`additional_constraints_for_highest_resolution` list in [/] {__file__}" ) @@ -632,8 +755,18 @@ def generate_constraints_no_providers(config_params: ConfigParams) -> None: help="Use uv instead of pip as packaging tool.", envvar="USE_UV", ) +@click.option( + "--allow-pre-releases", + is_flag=True, + default=False, + help="Let the provider pins use pre-release versions when those are newer than any final " + "release. Used when constraints are generated for a release candidate, whose providers are " + "only on PyPI as rc versions.", + envvar="ALLOW_PRE_RELEASES", +) def generate_constraints( airflow_constraints_mode: str, + allow_pre_releases: bool, constraints_github_repository: str, default_constraints_branch: str, github_actions: bool, @@ -642,6 +775,7 @@ def generate_constraints( ) -> None: config_params = ConfigParams( airflow_constraints_mode=airflow_constraints_mode, + allow_pre_releases=allow_pre_releases, constraints_github_repository=constraints_github_repository, default_constraints_branch=default_constraints_branch, github_actions=github_actions, diff --git a/scripts/in_container/run_provider_yaml_files_check.py b/scripts/in_container/run_provider_yaml_files_check.py index eb6f4421384fc..34e338a6a144e 100755 --- a/scripts/in_container/run_provider_yaml_files_check.py +++ b/scripts/in_container/run_provider_yaml_files_check.py @@ -379,6 +379,39 @@ def check_integration_duplicates(yaml_files: dict[str, dict]) -> tuple[int, int] return num_integrations, num_errors +@run_check("Checking remote-logging scheme duplicates") +def check_remote_logging_scheme_duplicates(yaml_files: dict[str, dict]) -> tuple[int, int]: + """Remote-logging ``scheme`` values must be globally unique across providers. + + The scheme in ``[logging] remote_base_log_folder`` selects which provider's + ``RemoteLogIO`` handler serves a user's task logs. ``ProvidersManager`` resolves a + collision silently — the first provider in alphabetical order wins and the rest are + dropped — so a duplicate scheme can quietly shadow another provider's log handler. + Fail here so the clash is caught before it ships instead of at runtime. + """ + num_errors = 0 + packages_by_scheme: dict[str, set[str]] = {} + for provider_data in yaml_files.values(): + package_name = provider_data["package-name"] + for entry in provider_data.get("remote-logging", []): + packages_by_scheme.setdefault(entry["scheme"], set()).add(package_name) + num_schemes = len(packages_by_scheme) + duplicates = [ + (scheme, ", ".join(sorted(packages))) + for scheme, packages in sorted(packages_by_scheme.items()) + if len(packages) > 1 + ] + if duplicates: + console.print( + "Duplicate remote-logging schemes found. Each scheme may be registered by only " + "one provider; otherwise ProvidersManager silently keeps the first provider " + "alphabetically and shadows the others. Please give each handler a unique scheme." + ) + errors.append(tabulate(duplicates, headers=["Remote-logging scheme", "Registered by"])) + num_errors += 1 + return num_schemes, num_errors + + @run_check("Checking completeness of list of {sensors, hooks, operators, triggers, bundles, toolsets}") def check_correctness_of_list_of_sensors_operators_hook_trigger_modules( yaml_files: dict[str, dict], @@ -1153,6 +1186,7 @@ def check_providers_have_all_documentation_files(yaml_files: dict[str, dict]): all_files_loaded = len(all_provider_files) == len(paths) check_integration_duplicates(all_parsed_yaml_files) + check_remote_logging_scheme_duplicates(all_parsed_yaml_files) check_duplicates_in_list_of_transfers(all_parsed_yaml_files) check_duplicates_in_integrations_names_of_hooks_sensors_operators(all_parsed_yaml_files) diff --git a/scripts/tests/ci/prek/test_extract_permissions.py b/scripts/tests/ci/prek/test_extract_permissions.py index 40be37dff0d28..2900c66f603b1 100644 --- a/scripts/tests/ci/prek/test_extract_permissions.py +++ b/scripts/tests/ci/prek/test_extract_permissions.py @@ -627,7 +627,8 @@ class TestExtractAllPermissions: """ @pytest.fixture(scope="class") - def all_entries(self) -> list[PermissionEntry]: + @classmethod + def all_entries(cls) -> list[PermissionEntry]: return extract_all_permissions(PUBLIC_ROUTES_DIR) def test_extracts_non_empty_result(self, all_entries): @@ -769,7 +770,8 @@ def test_clear_dag_runs_endpoint_prefix(self, all_entries): class TestRenderRst: @pytest.fixture(scope="class") - def rst_content(self) -> str: + @classmethod + def rst_content(cls) -> str: entries = extract_all_permissions(PUBLIC_ROUTES_DIR) return render_rst(entries) diff --git a/scripts/tests/ci/prek/test_validate_operators_init.py b/scripts/tests/ci/prek/test_validate_operators_init.py index 4546a173ca7c7..45302516bb109 100644 --- a/scripts/tests/ci/prek/test_validate_operators_init.py +++ b/scripts/tests/ci/prek/test_validate_operators_init.py @@ -23,6 +23,7 @@ import pytest import validate_operators_init +from rich.text import Text from validate_operators_init import ( _check_constructor_field_logic, _check_constructor_template_fields, @@ -131,6 +132,18 @@ class TestConstructorFieldLogic: def test_flags_logic_but_not_sanctioned_patterns(self, ctor_body: str, expected: int): assert _logic_findings(_operator_code(ctor_body), ["foo"]) == expected + def test_failure_message_explains_provision_check_exception(self): + code = _operator_code("self._validate(foo)\nself.foo = foo") + with validate_operators_init.console.capture() as capture: + assert _logic_findings(code, ["foo"]) == 1 + + output = " ".join(Text.from_ansi(capture.get()).plain.split()) + assert "value-dependent validation or transformation" in output + assert "whether an argument was passed" in output + assert "is None" in output + assert "is not None" in output + assert "https://github.com/apache/airflow/issues/70296#false-positives" in output + def test_name_in_parameter_default_is_not_the_field(self): # A field named like a module (e.g. "conf") used in a parameter default evaluates at # class-definition scope and must not be flagged. diff --git a/scripts/tests/ci/test_notify_uv_lock_conflicts.py b/scripts/tests/ci/test_notify_uv_lock_conflicts.py index eb9c41f322e56..f187d0c32243f 100644 --- a/scripts/tests/ci/test_notify_uv_lock_conflicts.py +++ b/scripts/tests/ci/test_notify_uv_lock_conflicts.py @@ -161,6 +161,35 @@ def test_previous_marker_with_different_sha_still_notifies(self, mod): assert entry["existing"] == {"id": "C1", "body": f"{mod.MARKER}\nolder notice for deadbee"} +class TestAvoidBacklink: + @pytest.mark.parametrize( + "url,expected", + [ + ( + "https://github.com/apache/airflow/pull/42", + "https://redirect.github.com/apache/airflow/pull/42", + ), + ( + "https://github.com/apache/airflow/commit/deadbeef", + "https://redirect.github.com/apache/airflow/commit/deadbeef", + ), + ], + ) + def test_rewrites_github_com(self, mod, url, expected): + assert mod.avoid_backlink(url) == expected + + def test_leaves_non_github_url_untouched(self, mod): + url = "https://example.com/pull/42" + assert mod.avoid_backlink(url) == url + + def test_only_rewrites_leading_occurrence(self, mod): + """Only the URL's own host is rewritten, not incidental later occurrences.""" + url = "https://github.com/apache/airflow/pull/42#see-https://github.com/other" + assert mod.avoid_backlink(url) == ( + "https://redirect.github.com/apache/airflow/pull/42#see-https://github.com/other" + ) + + class TestBuildBody: def test_includes_marker_source_and_instructions(self, mod): body = mod.build_body("[#42](https://example/pr/42)") diff --git a/scripts/tests/in_container/test_run_generate_constraints.py b/scripts/tests/in_container/test_run_generate_constraints.py index b2bd467faa383..99c7f4c5ee926 100644 --- a/scripts/tests/in_container/test_run_generate_constraints.py +++ b/scripts/tests/in_container/test_run_generate_constraints.py @@ -24,15 +24,38 @@ import run_generate_constraints as m -def _config(latest: Path, current: Path, constraints_dir: Path | None = None) -> SimpleNamespace: +def _config( + latest: Path, + current: Path, + constraints_dir: Path | None = None, + allow_pre_releases: bool = False, +) -> SimpleNamespace: return SimpleNamespace( latest_constraints_file=latest, current_constraints_file=current, constraints_dir=constraints_dir if constraints_dir is not None else latest.parent, python="3.10", + allow_pre_releases=allow_pre_releases, ) +def _file(*, yanked: bool = False, requires_python: str | None = ">=3.10") -> dict: + return {"yanked": yanked, "requires_python": requires_python} + + +class _FakeResponse: + def __init__(self, payload: dict, status_code: int = 200): + self._payload = payload + self.status_code = status_code + + def json(self) -> dict: + return self._payload + + def raise_for_status(self) -> None: + if self.status_code >= 400: + raise AssertionError(f"Unexpected status {self.status_code}") + + class TestReadProviderVersionsFromConstraints: def test_extracts_only_providers_and_strips_markers(self, tmp_path): constraints = tmp_path / "constraints.txt" @@ -82,6 +105,16 @@ def test_slack_channel_is_overridable_via_env(self, tmp_path, monkeypatch): payload = json.loads((tmp_path / "provider-downgrade-slack-message.json").read_text()) assert payload["channel"] == "some-other-channel" + def test_skips_the_check_for_a_release_candidate(self, tmp_path): + latest = tmp_path / "latest.txt" + current = tmp_path / "current.txt" + latest.write_text("apache-airflow-providers-amazon==8.0.0\n") + current.write_text("apache-airflow-providers-amazon==7.5.0\n") + + m.check_providers_not_downgraded(_config(latest, current, allow_pre_releases=True)) + + assert not (tmp_path / "provider-downgrade-slack-message.json").exists() + def test_passes_when_no_provider_is_downgraded(self, tmp_path): latest = tmp_path / "latest.txt" current = tmp_path / "current.txt" @@ -109,3 +142,133 @@ def test_skips_uncomparable_versions_without_error(self, tmp_path): latest.write_text("apache-airflow-providers-amazon==not-a-version\n") current.write_text("apache-airflow-providers-amazon==also-bad\n") m.check_providers_not_downgraded(_config(latest, current)) + + +class TestFindNewestVersionInPypi: + @pytest.mark.parametrize( + "releases, allow_pre_releases, expected", + [ + pytest.param( + {"1.0.0": [_file()], "1.1.0": [_file()]}, + False, + "1.1.0", + id="newest-final-release", + ), + pytest.param( + {"1.1.0": [_file()], "1.2.0rc1": [_file()]}, + True, + "1.2.0rc1", + id="candidate-newer-than-the-release-wins-when-allowed", + ), + pytest.param( + {"1.1.0": [_file()], "1.2.0rc1": [_file()]}, + False, + "1.1.0", + id="candidate-is-ignored-when-not-allowed", + ), + pytest.param( + {"1.1.0rc1": [_file()], "1.1.0": [_file()]}, + True, + "1.1.0", + id="superseded-candidate-loses-to-its-release", + ), + pytest.param( + {"1.0.0": [_file()], "1.1.0": [_file(yanked=True)]}, + False, + "1.0.0", + id="yanked-version-is-passed-over", + ), + pytest.param( + {"1.0.0": [_file()], "1.1.0": []}, + False, + "1.0.0", + id="version-with-no-files-is-passed-over", + ), + pytest.param( + {"1.0.0": [_file()], "1.1.0": [_file(requires_python=">=3.12")]}, + False, + "1.0.0", + id="version-excluded-by-requires-python-is-passed-over", + ), + pytest.param( + {"1.0.0": [_file()], "not-a-version": [_file()]}, + False, + "1.0.0", + id="unparseable-version-is-passed-over", + ), + pytest.param( + {"1.0.0": [_file(requires_python=None)]}, + False, + "1.0.0", + id="file-without-requires-python-is-installable", + ), + pytest.param( + {"1.0.0": [_file(requires_python="not-a-specifier")]}, + False, + "1.0.0", + id="file-with-broken-requires-python-is-not-excluded", + ), + pytest.param({"1.0.0": [_file(yanked=True)]}, False, None, id="nothing-installable-is-unpinned"), + pytest.param({"1.0.0rc1": [_file()]}, False, None, id="only-a-candidate-is-unpinned"), + ], + ) + def test_picks_the_newest_installable_version(self, monkeypatch, releases, allow_pre_releases, expected): + monkeypatch.setattr(m.requests, "get", lambda url, timeout: _FakeResponse({"releases": releases})) + + newest = m.find_newest_version_in_pypi("apache-airflow-providers-amazon", "3.10", allow_pre_releases) + + assert newest == expected + + def test_unpublished_distribution_is_unpinned(self, monkeypatch): + monkeypatch.setattr(m.requests, "get", lambda url, timeout: _FakeResponse({}, status_code=404)) + + assert m.find_newest_version_in_pypi("apache-airflow-providers-brand-new", "3.10", True) is None + + +class TestBuildPinnedProviderRequirements: + """Providers are pinned at what PyPI holds rather than left for the resolution to pick.""" + + def test_every_provider_is_pinned_to_its_newest_pypi_version(self, monkeypatch): + monkeypatch.setattr( + m, + "get_all_active_provider_distributions", + lambda python_version=None: [ + "apache-airflow-providers-amazon", + "apache-airflow-providers-cncf-kubernetes", + "apache-airflow-providers-brand-new", + ], + ) + newest = { + "apache-airflow-providers-amazon": "9.0.0rc1", + "apache-airflow-providers-cncf-kubernetes": "10.20.0", + "apache-airflow-providers-brand-new": None, + } + monkeypatch.setattr( + m, "find_newest_version_in_pypi", lambda dist, python_version, allow_pre_releases: newest[dist] + ) + + assert m.build_pinned_provider_requirements("3.10", True) == [ + "apache-airflow-providers-amazon==9.0.0rc1", + "apache-airflow-providers-cncf-kubernetes==10.20.0", + ] + + def test_the_python_version_and_the_pre_release_choice_reach_the_lookup(self, monkeypatch): + """Providers excluded on a Python version must not be named, nor pinned to a version it cannot install.""" + seen_for_distributions: list[str | None] = [] + seen_for_lookup: list[tuple[str, bool]] = [] + + def fake_distributions(python_version=None): + seen_for_distributions.append(python_version) + return ["apache-airflow-providers-amazon"] + + def fake_lookup(distribution, python_version, allow_pre_releases): + seen_for_lookup.append((python_version, allow_pre_releases)) + return "9.0.0" + + monkeypatch.setattr(m, "get_all_active_provider_distributions", fake_distributions) + monkeypatch.setattr(m, "find_newest_version_in_pypi", fake_lookup) + + m.build_pinned_provider_requirements("3.14", False) + + assert seen_for_distributions == ["3.14"] + assert seen_for_lookup == [("3.14", False)] diff --git a/shared/configuration/src/airflow_shared/configuration/parser.py b/shared/configuration/src/airflow_shared/configuration/parser.py index 2cfb946ad32f0..8c9e579bf996e 100644 --- a/shared/configuration/src/airflow_shared/configuration/parser.py +++ b/shared/configuration/src/airflow_shared/configuration/parser.py @@ -77,6 +77,34 @@ def _collect_kwarg_env_vars(prefix: str) -> dict[str, str]: ConfigSectionSourcesType = dict[str, str | tuple[str, str]] ConfigSourcesType = dict[str, ConfigSectionSourcesType] ENV_VAR_PREFIX = "AIRFLOW__" +# Separates the team name from the base section name in a team scoped config file section. +TEAM_SECTION_SEPARATOR = "=" + + +def team_section_name(team_name: str, section: str) -> str: + """ + Build the config file section name that holds the team scoped overrides of ``section``. + + :param team_name: name of the team the overrides belong to + :param section: base section name that is being overridden + :return: the team scoped section name, e.g. ``team_a=celery`` + """ + return f"{team_name}{TEAM_SECTION_SEPARATOR}{section}" + + +def base_section_name(section: str) -> str: + """ + Return the base section name of a possibly team scoped config file section. + + Team scoped sections are built by :func:`team_section_name`. Base section names never contain + the separator, so the name is split on the last one - that way the base section is recovered + even for a team name that contains the separator itself. + + :param section: section name, either a base one or a team scoped one + :return: the base section name, which is ``section`` itself when it is not team scoped + """ + _, separator, base_section = section.rpartition(TEAM_SECTION_SEPARATOR) + return base_section if separator else section if TYPE_CHECKING: @@ -562,6 +590,68 @@ def sensitive_config_values(self) -> set[tuple[str, str]]: sensitive.update(depr_section, depr_option) return sensitive + def _names_sensitive_team_env_var(self, env_var: str) -> bool: + """ + Check whether an environment variable name is a team scoped override of a sensitive option. + + Team scoped variables are named ``AIRFLOW_____
__`` (see + :meth:`_env_var_name`). A team name may contain underscores itself, so the team name is not + parsed out of the variable name; the name is matched against the tail that each option + registered as sensitive contributes instead. The ``_CMD`` / ``_SECRET`` fallbacks are + matched too, because - unlike for a base section - they are never resolved into their value. + + :param env_var: environment variable name + :return: True if the variable holds a team scoped value of an option registered as sensitive + """ + env_var = env_var.upper() + if not env_var.startswith(ENV_VAR_PREFIX): + return False + # Every tail matched below starts with the ``___`` separating the team name from the + # section, so a name not containing it cannot be a team scoped one. + if "___" not in env_var: + return False + for section, key in self.sensitive_config_values: + option_tail = self._env_var_name(section, key).removeprefix(ENV_VAR_PREFIX) + for tail in (f"___{option_tail}", f"___{option_tail}_CMD", f"___{option_tail}_SECRET"): + # The team name sits between the prefix and the tail, so it must not be empty. + if env_var.endswith(tail) and len(env_var) > len(ENV_VAR_PREFIX) + len(tail): + return True + return False + + def is_sensitive_option(self, section: str, key: str) -> bool: + """ + Check whether the value of ``key`` in ``section`` is registered as sensitive. + + Options are registered as sensitive under their base section name, while a team scoped + override of the very same option is held by a ``=
`` config file section + or by an ``AIRFLOW_____
__`` environment variable. Both of those + spellings are resolved back to the base option here, so that a team scoped value is treated + exactly like the base one. A name that does not resolve to a registered option is not + sensitive - so this only ever recognises more options as sensitive, never fewer. + + :param section: section name, either a base one or a team scoped one + :param key: option name + :return: True if the value of the option should be treated as sensitive + """ + section = section.lower() + key = key.lower() + if (section, key) in self.sensitive_config_values: + return True + base_section = base_section_name(section) + if base_section != section: + if (base_section, key) in self.sensitive_config_values: + return True + # A team scoped ``_cmd`` / ``_secret`` fallback is not resolved into its value, so it + # stays in the output as configured and has to be recognised on its own. + for fallback_suffix in ("_cmd", "_secret"): + if not key.endswith(fallback_suffix): + continue + if (base_section, key.removesuffix(fallback_suffix)) in self.sensitive_config_values: + return True + # A team scoped environment variable is reported under the section and key its name splits + # into, which is neither the base nor the team scoped section name. + return self._names_sensitive_team_env_var(self._env_var_name(section, key)) + def _update_defaults_from_string(self, config_string: str) -> None: """ Update the defaults in _default_values based on values in config_string ("ini" format). @@ -788,8 +878,11 @@ def _include_envs( log.warning("Ignoring unknown env var '%s'", env_var) continue if not display_sensitive and env_var != self._env_var_name("core", "unit_test_mode"): + if self._names_sensitive_team_env_var(env_var): + # Covers the cmd/secret variants too; see is_sensitive_option. + opt = "< hidden >" # Don't hide cmd/secret values here - if not env_var.lower().endswith(("cmd", "secret")): + elif not env_var.lower().endswith(("cmd", "secret")): if (section, key) in self.sensitive_config_values: opt = "< hidden >" elif raw: @@ -1121,7 +1214,7 @@ def _get_option_from_config_file( ) -> str | ValueNotFound: """Get config option from config file.""" if team_name := kwargs.get("team_name", None): - section = f"{team_name}={section}" + section = team_section_name(team_name, section) # since this is the last lookup that supports team_name, pop it kwargs.pop("team_name") if super().has_option(section, key): @@ -1792,15 +1885,19 @@ def as_dict( if not display_sensitive: # This ensures the ones from config file is hidden too # if they are not provided through env, cmd and secret + # The collected options are walked (rather than the registered sensitive ones) so that + # team scoped sections are covered as well - they are named after the team, not after + # the base section the option is registered under. hidden = "< hidden >" - for section, key in self.sensitive_config_values: - if config_sources.get(section): - if config_sources[section].get(key, None): - if display_source: - source = config_sources[section][key][1] - config_sources[section][key] = (hidden, source) - else: - config_sources[section][key] = hidden + for section, options in config_sources.items(): + for key, value in list(options.items()): + if not value or not self.is_sensitive_option(section, key): + continue + if display_source: + source = value[1] + options[key] = (hidden, source) + else: + options[key] = hidden return config_sources @@ -1877,7 +1974,7 @@ def getsection(self, section: str, team_name: str | None = None) -> ConfigOption :param team_name: optional team name for team-specific configuration lookup """ # Handle team-specific section lookup for config file - config_section = f"{team_name}={section}" if team_name else section + config_section = team_section_name(team_name, section) if team_name else section if not self.has_section(config_section) and not self._has_section_in_any_defaults(config_section): return None @@ -2032,10 +2129,7 @@ def write( # type: ignore[override] section_to_write=section_to_write, sources_dict=sources_dict, ) - is_sensitive = ( - section_to_write.lower(), - option.lower(), - ) in self.sensitive_config_values + is_sensitive = self.is_sensitive_option(section_to_write, option) self._write_value( file=file, option=option, diff --git a/shared/configuration/tests/configuration/test_parser.py b/shared/configuration/tests/configuration/test_parser.py index 6c1b5c1989f43..6e93cf70eff75 100644 --- a/shared/configuration/tests/configuration/test_parser.py +++ b/shared/configuration/tests/configuration/test_parser.py @@ -34,6 +34,7 @@ from airflow_shared.configuration.exceptions import AirflowConfigException from airflow_shared.configuration.parser import ( AirflowConfigParser as _SharedAirflowConfigParser, + base_section_name, configure_parser_from_configuration_description, ) @@ -1124,6 +1125,148 @@ def test_team_env_var_format(self): ): assert test_conf.get("my_section", "my_key", team_name="my_team") == "team_value" + @pytest.mark.parametrize( + ("section", "expected"), + [ + pytest.param("celery", "celery", id="base_section"), + pytest.param("team_a=celery", "celery", id="team_scoped_section"), + pytest.param("team-a=celery", "celery", id="team_name_with_dash"), + pytest.param("a=team=celery", "celery", id="separator_inside_team_name"), + ], + ) + def test_base_section_name(self, section, expected): + """The base section name is recovered from a team scoped section name.""" + assert base_section_name(section) == expected + + def test_is_sensitive_option_resolves_team_scoped_names(self): + """A team scoped spelling of a registered sensitive option is recognised as sensitive.""" + test_conf = AirflowConfigParser() + test_conf.sensitive_config_values.add(("test", "sensitive_key")) + + assert test_conf.is_sensitive_option("test", "sensitive_key") + assert test_conf.is_sensitive_option("team_a=test", "sensitive_key") + assert test_conf.is_sensitive_option("team-a=test", "sensitive_key") + assert test_conf.is_sensitive_option("a=team=test", "sensitive_key") + # The section and key an AIRFLOW_____
__ variable name splits into. + assert test_conf.is_sensitive_option("team_a", "_test__sensitive_key") + + # An option that is not registered as sensitive stays readable under any spelling. + assert not test_conf.is_sensitive_option("test", "key1") + assert not test_conf.is_sensitive_option("team_a=test", "key1") + assert not test_conf.is_sensitive_option("team_a", "_test__key1") + # A team name matching a registered option does not make the section sensitive either. + assert not test_conf.is_sensitive_option("sensitive_key=test", "key1") + + def test_team_scoped_sensitive_value_is_hidden_in_as_dict(self): + """A sensitive option set in a [team=section] section is hidden like the base option is.""" + test_conf = AirflowConfigParser() + test_conf.read_string( + textwrap.dedent( + """\ + [test] + sensitive_key = global_value + + [team_a=test] + sensitive_key = team_a_value + key1 = team_a_key1_value + """ + ) + ) + test_conf.sensitive_config_values.add(("test", "sensitive_key")) + + as_dict = test_conf.as_dict(display_sensitive=False) + + assert as_dict["team_a=test"]["sensitive_key"] == "< hidden >" + # The base option keeps being hidden, and an option that is not registered as sensitive + # keeps being readable. + assert as_dict["test"]["sensitive_key"] == "< hidden >" + assert as_dict["team_a=test"]["key1"] == "team_a_key1_value" + + # display_source keeps reporting where the value came from + as_dict_with_source = test_conf.as_dict(display_sensitive=False, display_source=True) + assert as_dict_with_source["team_a=test"]["sensitive_key"] == ("< hidden >", "airflow.cfg") + + # display_sensitive=True still returns the real values + as_dict_sensitive = test_conf.as_dict(display_sensitive=True) + assert as_dict_sensitive["team_a=test"]["sensitive_key"] == "team_a_value" + assert as_dict_sensitive["test"]["sensitive_key"] == "global_value" + + def test_team_scoped_sensitive_cmd_and_secret_fallbacks_are_hidden_in_as_dict(self): + """The _cmd / _secret fallbacks of a team scoped option are not resolved, so they are hidden.""" + test_conf = AirflowConfigParser() + test_conf.read_string( + textwrap.dedent( + """\ + [team_a=test] + sensitive_key_cmd = echo -n team_a_value + sensitive_key_secret = team_a/secret/path + """ + ) + ) + test_conf.sensitive_config_values.add(("test", "sensitive_key")) + + as_dict = test_conf.as_dict(display_sensitive=False) + + assert as_dict["team_a=test"]["sensitive_key_cmd"] == "< hidden >" + assert as_dict["team_a=test"]["sensitive_key_secret"] == "< hidden >" + + def test_team_scoped_sensitive_env_var_is_hidden_in_as_dict(self): + """A sensitive option set by a team scoped env var is hidden like the base option is.""" + test_conf = AirflowConfigParser() + test_conf.sensitive_config_values.add(("test", "sensitive_key")) + + with patch.dict( + os.environ, + { + "AIRFLOW__TEAM_A___TEST__SENSITIVE_KEY": "team_a_value", + "AIRFLOW__TEAM_A___TEST__SENSITIVE_KEY_CMD": "echo -n team_a_cmd_value", + "AIRFLOW__TEAM_A___TEST__KEY1": "team_a_key1_value", + }, + ): + hidden_values = self._collected_values(test_conf.as_dict(display_sensitive=False)) + sensitive_values = self._collected_values(test_conf.as_dict(display_sensitive=True)) + + assert "team_a_value" not in hidden_values + assert "echo -n team_a_cmd_value" not in hidden_values + assert "< hidden >" in hidden_values + assert "team_a_key1_value" in hidden_values + # display_sensitive=True still returns the real value. + assert "team_a_value" in sensitive_values + + def test_team_scoped_sensitive_value_is_hidden_by_write(self): + """A sensitive option set in a [team=section] section is hidden when writing the config out.""" + test_conf = AirflowConfigParser() + test_conf.read_string( + textwrap.dedent( + """\ + [team_a=test] + sensitive_key = team_a_value + key1 = team_a_key1_value + """ + ) + ) + test_conf.sensitive_config_values.add(("test", "sensitive_key")) + + file = StringIO() + test_conf.write( + file, + include_descriptions=False, + include_examples=False, + include_sources=False, + show_values=True, + hide_sensitive=True, + ) + written = file.getvalue() + + assert "team_a_value" not in written + assert "sensitive_key = < hidden >" in written + assert "key1 = team_a_key1_value" in written + + @staticmethod + def _collected_values(as_dict): + """Flatten the values of an ``as_dict`` result, whatever section they were reported under.""" + return {value for options in as_dict.values() for value in options.values()} + @pytest.mark.parametrize( "populate_caches", [ diff --git a/task-sdk/docs/api.rst b/task-sdk/docs/api.rst index 51aa275b90bb0..2c9bbe3f4db80 100644 --- a/task-sdk/docs/api.rst +++ b/task-sdk/docs/api.rst @@ -116,6 +116,10 @@ Deadline Alerts .. autoclass:: airflow.sdk.DeadlineReference +.. autoclass:: airflow.sdk.BaseDeadlineReference + +.. autofunction:: airflow.sdk.deadline_reference + Connections & Variables ----------------------- .. autoapiclass:: airflow.sdk.Connection diff --git a/task-sdk/pyproject.toml b/task-sdk/pyproject.toml index 6c540ee3bc9b6..bb8f7e2c3f4d4 100644 --- a/task-sdk/pyproject.toml +++ b/task-sdk/pyproject.toml @@ -224,18 +224,9 @@ dev = [ "apache-airflow-providers-common-sql", "apache-airflow-providers-standard", "apache-airflow-devel-common", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - "pandas>=2.1.2,<3; python_version <\"3.13\"", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - "pandas>=2.2.3,<3; python_version >=\"3.13\" and python_version <\"3.14\"", - # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not - # round trip; capped until pandas 3 support lands. - # Tracked at https://github.com/apache/airflow/pull/70558 - "pandas>=2.3.3,<3; python_version >=\"3.14\"" + "pandas>=2.1.2; python_version <\"3.13\"", + "pandas>=2.2.3; python_version >=\"3.13\" and python_version <\"3.14\"", + "pandas>=2.3.3; python_version >=\"3.14\"" ] docs = [ "apache-airflow-devel-common[docs]", diff --git a/task-sdk/src/airflow/sdk/__init__.py b/task-sdk/src/airflow/sdk/__init__.py index 807019eade80e..6cff747fccc94 100644 --- a/task-sdk/src/airflow/sdk/__init__.py +++ b/task-sdk/src/airflow/sdk/__init__.py @@ -31,6 +31,7 @@ "AsyncCallback", "BaseAsyncOperator", "BaseBranchOperator", + "BaseDeadlineReference", "BaseHook", "BaseNotifier", "BaseOperator", @@ -105,6 +106,7 @@ "conf", "cross_downstream", "dag", + "deadline_reference", "get_current_context", "get_parsing_context", "literal", @@ -151,7 +153,12 @@ from airflow.sdk.definitions.connection import Connection from airflow.sdk.definitions.context import Context, get_current_context, get_parsing_context from airflow.sdk.definitions.dag import DAG, dag - from airflow.sdk.definitions.deadline import DeadlineAlert, DeadlineReference + from airflow.sdk.definitions.deadline import ( + BaseDeadlineReference, + DeadlineAlert, + DeadlineReference, + deadline_reference, + ) from airflow.sdk.definitions.decorators import result, setup, task, teardown from airflow.sdk.definitions.decorators.task_group import task_group from airflow.sdk.definitions.edges import EdgeModifier, Label @@ -234,6 +241,7 @@ "AsyncCallback": ".definitions.callback", "BaseAsyncOperator": ".bases.operator", "BaseBranchOperator": ".bases.branch", + "BaseDeadlineReference": ".definitions.deadline", "BaseHook": ".bases.hook", "BaseNotifier": ".bases.notifier", "BaseOperator": ".bases.operator", @@ -308,6 +316,7 @@ "conf": ".configuration", "cross_downstream": ".bases.operator", "dag": ".definitions.dag", + "deadline_reference": ".definitions.deadline", "NEVER_EXPIRE": ".execution_time.context", "get_current_context": ".definitions.context", "get_parsing_context": ".definitions.context", diff --git a/task-sdk/src/airflow/sdk/definitions/callback.py b/task-sdk/src/airflow/sdk/definitions/callback.py index 4280a43513d24..355294303ab2a 100644 --- a/task-sdk/src/airflow/sdk/definitions/callback.py +++ b/task-sdk/src/airflow/sdk/definitions/callback.py @@ -28,6 +28,18 @@ log = structlog.getLogger(__name__) +class _SerializedCallbackPath(str): + """ + A callback path which was read back from serialized data. + + See :meth:`Callback.get_callback_path` for what the marker buys. + + :meta private: + """ + + __slots__ = () + + class Callback(ABC): """ Base class for Deadline Alert callbacks. @@ -66,6 +78,14 @@ def get_callback_path(cls, _callback: str | Callable) -> str: stripped_callback = _callback.strip() + if isinstance(_callback, _SerializedCallbackPath): + # This path was already checked when the Callback it belongs to was created, so + # rebuilding that Callback keeps it as it is. Reconstruction happens in components + # which never call the callback themselves, and resolving the path there is neither + # needed nor wanted: the module it names is not necessarily importable in that + # process, and importing it has no bearing on the path we return either way. + return stripped_callback + try: # The provided callback is a string which appears to be a valid dotpath, attempt to import it. callback = import_string(stripped_callback) @@ -95,6 +115,8 @@ def verify_callable(cls, callback: Callable): @classmethod def deserialize(cls, data: dict, version): path = data.pop("path") + if isinstance(path, str): + path = _SerializedCallbackPath(path) return cls(callback_callable=path, **data) @classmethod diff --git a/task-sdk/src/airflow/sdk/definitions/dag.py b/task-sdk/src/airflow/sdk/definitions/dag.py index cadfe6978300e..633feccb0483a 100644 --- a/task-sdk/src/airflow/sdk/definitions/dag.py +++ b/task-sdk/src/airflow/sdk/definitions/dag.py @@ -392,6 +392,9 @@ class DAG: "{'role1': {'can_read'}, 'role2': {'can_read', 'can_edit', 'can_delete'}}" or it can specify the resource name if there is a DAGs Run resource, e.g., "{'role1': {'DAG Runs': {'can_create'}}, 'role2': {'DAGs': {'can_read', 'can_edit', 'can_delete'}}" + Deprecated, and only read when the configured auth manager is the FAB auth manager. Under any + other auth manager this argument is ignored and grants no access. See + https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html :param is_paused_upon_creation: Specifies if the dag is paused when created for the first time. If the dag exists already, this flag will be ignored. If this optional parameter is not specified, the global config setting will be used. @@ -588,7 +591,9 @@ def __attrs_post_init__(self): self.default_args["end_date"] = timezone.convert_to_utc(end_date) if self.access_control is not None: warnings.warn( - "The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html", + "DAG.access_control is deprecated and is only read when the configured auth manager is the " + "FAB auth manager; under any other auth manager it is ignored and grants no access. See " + "https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html", RemovedInAirflow4Warning, stacklevel=2, ) diff --git a/task-sdk/src/airflow/sdk/definitions/deadline.py b/task-sdk/src/airflow/sdk/definitions/deadline.py index ab01a9213aa5f..6b895c2164e23 100644 --- a/task-sdk/src/airflow/sdk/definitions/deadline.py +++ b/task-sdk/src/airflow/sdk/definitions/deadline.py @@ -42,11 +42,11 @@ class BaseDeadlineReference(ABC): """ Base class for all Deadline Reference implementations. - This is a lightweight SDK class for DAG authoring. It only handles serialization. - The actual evaluation logic (_evaluate_with) is in Core's SerializedReferenceModels. + This is a lightweight SDK class for Dag authoring. It only handles serialization. + The actual evaluation logic (``_evaluate_with``) is in Core's ``SerializedReferenceModels``. For custom deadline references, users should inherit from this class and implement - _evaluate_with() with deferred Core imports (imports inside the method body). + ``_evaluate_with()`` with deferred Core imports (imports inside the method body). """ @property diff --git a/task-sdk/src/airflow/sdk/execution_time/context.py b/task-sdk/src/airflow/sdk/execution_time/context.py index f0c982b32ea0b..7ddc6edf1a3aa 100644 --- a/task-sdk/src/airflow/sdk/execution_time/context.py +++ b/task-sdk/src/airflow/sdk/execution_time/context.py @@ -325,7 +325,12 @@ def _mask_and_deserialize_variable(raw: str, key: str, deserialize_json: bool) - if isinstance(val, str): mask_secret(val, key) elif isinstance(val, dict): + # Masked by the dict's own inner key names, which is what ``add_mask`` uses. mask_secret(val) + elif isinstance(val, list): + # Pass the Variable's key so list elements inherit the Variable's sensitivity + # instead of being added to the global mask patterns. + mask_secret(val, key) return val diff --git a/task-sdk/src/airflow/sdk/execution_time/schema/AGENTS.md b/task-sdk/src/airflow/sdk/execution_time/schema/AGENTS.md index 0d844efe7ac05..84c821bad2bbc 100644 --- a/task-sdk/src/airflow/sdk/execution_time/schema/AGENTS.md +++ b/task-sdk/src/airflow/sdk/execution_time/schema/AGENTS.md @@ -83,11 +83,14 @@ head shape *is* the schema for the new body. ) ``` -3. Reference the new `VersionChange` from the bundle in - `versions/__init__.py`: +3. Reference the new `VersionChange` from the bundle returned by + `get_bundle()` in `versions/__init__.py`: ```python - Version("2026-06-16", AddRetryDelay, AddSentryTraceField), + return VersionBundle( + HeadVersion(), + Version("2026-06-16", AddRetryDelay, AddSentryTraceField), + ) ``` 4. The `generate-supervisor-schemas-snapshot` prek hook will diff --git a/task-sdk/src/airflow/sdk/execution_time/supervisor.py b/task-sdk/src/airflow/sdk/execution_time/supervisor.py index 87311f02da7a1..9758d2b18d7c8 100644 --- a/task-sdk/src/airflow/sdk/execution_time/supervisor.py +++ b/task-sdk/src/airflow/sdk/execution_time/supervisor.py @@ -418,8 +418,6 @@ def _fork_main( - Catch un-handled exceptions and attempt to show _something_ in case of error - Finally, run the actual task runner code (``target`` argument, defaults to ``.task_runner:main`) """ - # TODO: Make this process a session leader - # Store original stderr for last-chance exception handling last_chance_stderr = _get_last_chance_stderr() @@ -673,6 +671,9 @@ class WatchedSubprocess: subprocess_logs_to_stdout: bool = False """Duplicate log messages to stdout, or only send them to ``self.process_log``.""" + _new_process_group: bool = False + """Whether the child was placed in its own process group at fork time (see ``start``).""" + start_time: float = attrs.field(factory=time.monotonic) """The start time of the child process.""" @@ -683,6 +684,7 @@ def start( target: Callable[[], None] = _subprocess_main, logger: FilteringBoundLogger | None = None, use_exec: bool = False, + new_process_group: bool = False, **constructor_kwargs, ) -> Self: """ @@ -694,6 +696,12 @@ def start( ``target`` is rehydrated in the exec'd child from its ``module:qualname``, so any importable entry point (task execution, DAG processor, triggerer) is supported. + :param new_process_group: If True, place the child in its own process + group (PGID == its PID, like + ``airflow.utils.process_utils.set_new_process_group``) so signals + can be delivered to the child's whole process tree via + ``os.killpg``. Task execution opts in; DAG processor and triggerer + keep the supervisor's process group and are signalled directly. """ if use_exec and "<" in getattr(target, "__qualname__", "<"): # Closures/lambdas (```` / ```` in the qualname) and @@ -711,6 +719,19 @@ def start( pid = os.fork() if pid == 0: + if new_process_group: + # Put the task-runner into its own process group so its PGID + # equals its own PID. The supervisor can then deliver signals + # to the whole tree via os.killpg(), reaching every subprocess + # the task-runner spawned (e.g. venv children from + # PythonVirtualenvOperator). Without this, a SIGTERM from + # kill() only hits the task-runner and any Popen children are + # reparented to PID 1 and leak as orphans. Also set from the + # parent below so the group exists no matter which side of the + # fork runs first. See issue #65505. + with suppress(OSError): + os.setpgid(0, 0) + # Close and delete of the parent end of the sockets. cls._close_unused_sockets(read_requests, read_stdout, read_stderr, read_logs) @@ -762,6 +783,15 @@ def start( # do then _THINGS GET WEIRD_.. (Normally `_fork_main` itself will `_exit()` so we never get here) os._exit(124) + if new_process_group: + # Mirror of the child-side setpgid, so the group is guaranteed to + # exist once start() returns. Without this, kill() invoked before + # the child is first scheduled (e.g. task_instances.start() + # failing synchronously in _on_child_started) would resolve the + # child's PGID to the supervisor's own group and killpg it. + with suppress(OSError): + os.setpgid(pid, pid) + # Close the remaining parent-end of the sockets we've passed to the child via fork. We still have the # other end of the pair open cls._close_unused_sockets(child_stdout, child_stderr, child_logs) @@ -773,6 +803,7 @@ def start( process=PsutilTracker(psutil.Process(pid)), process_log=logger, start_time=time.monotonic(), + new_process_group=new_process_group, **constructor_kwargs, ) @@ -1007,6 +1038,30 @@ def _cleanup_open_sockets(self): self.selector.close() self.stdin.close() + def _signal_subprocess(self, sig: signal.Signals) -> None: + """ + Deliver ``sig`` to the child process, or to its whole process group when it has its own. + + When ``new_process_group`` was set at ``start()`` time, the signal is sent with + ``os.killpg`` so subprocesses spawned by the child (venv children, bash shells, etc.) + are reached too (see issue #65505). Falls back to signalling the child PID alone when + the group cannot be resolved or signalled -- and, critically, when the child still + shares the supervisor's own process group (``setpgid`` failed), because ``killpg`` on + our own group would signal the supervisor itself and its siblings. + """ + if self._new_process_group: + try: + pgid = os.getpgid(self._process.pid) + except (ProcessLookupError, PermissionError): + pgid = None + if pgid is not None and pgid != os.getpgid(0): + try: + os.killpg(pgid, sig) + return + except (ProcessLookupError, PermissionError): + pass + self._process.send_signal(sig) + def kill( self, signal_to_send: signal.Signals = signal.SIGINT, @@ -1036,7 +1091,7 @@ def kill( for sig in escalation_path: try: - self._process.send_signal(sig) + self._signal_subprocess(sig) start = time.monotonic() end = start + escalation_delay @@ -1360,7 +1415,13 @@ def start( # type: ignore[override] # infrastructure; keep bare fork for those. use_exec = target is _subprocess_main and _should_use_exec() proc: Self = super().start( - id=what.id, client=client, target=target, logger=logger, use_exec=use_exec, **kwargs + id=what.id, + client=client, + target=target, + logger=logger, + use_exec=use_exec, + new_process_group=True, + **kwargs, ) # Tell the task process what it needs to do! proc._on_child_started( diff --git a/task-sdk/src/airflow/sdk/execution_time/task_runner.py b/task-sdk/src/airflow/sdk/execution_time/task_runner.py index 31d0831bbadec..1ce848128d663 100644 --- a/task-sdk/src/airflow/sdk/execution_time/task_runner.py +++ b/task-sdk/src/airflow/sdk/execution_time/task_runner.py @@ -73,9 +73,12 @@ from airflow.sdk.definitions.param import process_params from airflow.sdk.exceptions import ( AirflowException, + AirflowFailException, AirflowInactiveAssetInInletOrOutletException, AirflowRescheduleException, AirflowRuntimeError, + AirflowSensorTimeout, + AirflowTaskTerminated, AirflowTaskTimeout, ErrorType, TaskAwaitingInput, @@ -238,6 +241,9 @@ class RuntimeTaskInstance(TaskInstance): _terminal_state_send_failed: bool = False """True when the supervisor IPC send for a non-success terminal state raised; signals main() to sys.exit(1) after finalize() so the supervisor doesn't misclassify the run as SUCCESS via exit code 0.""" + _failure_metrics_emitted: bool = False + """True once the failure counters have been recorded, so a second pass through the failure path (one attempt raised part-way through) does not count the same failure twice.""" + _ti_context_from_server: Annotated[TIRunContext | None, Field(repr=False)] = None """The Task Instance context from the API server, if any.""" @@ -1524,22 +1530,17 @@ def _await_input_task( return msg, state -@Sentry.enrich_errors -@detail_span("run") -def run( +def _run_task_and_map_outcome( ti: RuntimeTaskInstance, context: Context, log: Logger, ) -> tuple[TaskInstanceState, ToSupervisor | None, BaseException | None]: - """Run the task in this process.""" + """Execute the task and map its outcome -- success or a handled exception -- to a message and state.""" import signal from airflow.sdk.exceptions import ( - AirflowFailException, AirflowRescheduleException, - AirflowSensorTimeout, AirflowSkipException, - AirflowTaskTerminated, DagRunTriggerException, DownstreamTasksSkipped, TaskAwaitingInput, @@ -1565,9 +1566,6 @@ def _on_term(signum, frame): state: TaskInstanceState | None = None error: BaseException | None = None - stats_tags = ti.stats_tags - stats.incr("ti.start", tags=stats_tags) - try: # First, clear the xcom data sent from server if ti._ti_context_from_server and (keys_to_delete := ti._ti_context_from_server.xcom_keys_to_clear): @@ -1691,6 +1689,32 @@ def _on_term(signum, frame): log.info("::group::Post Execute") msg, state = _handle_current_task_failed(ti, e, log, context) error = e + + return state, msg, error + + +@Sentry.enrich_errors +@detail_span("run") +def run( + ti: RuntimeTaskInstance, + context: Context, + log: Logger, +) -> tuple[TaskInstanceState, ToSupervisor | None, BaseException | None]: + """Run the task in this process.""" + msg: ToSupervisor | None = None + state: TaskInstanceState | None = None + error: BaseException | None = None + + stats_tags = ti.stats_tags + stats.incr("ti.start", tags=stats_tags) + + try: + state, msg, error = _run_task_and_map_outcome(ti, context, log) + except Exception as e: + # Python does not offer an exception raised inside an ``except`` clause to that + # clause's siblings, so a handler that fails would otherwise escape entirely -- + # skipping the retry decision and every callback in ``finalize()``. + msg, state, error = _handle_handler_failure(ti, e, log, context) finally: # `state` may still be unset if an exception handler above raised before # binding it @@ -1798,6 +1822,46 @@ def _evaluate_retry_policy( return None +def _handle_handler_failure( + ti: RuntimeTaskInstance, exception: Exception, log: Logger, context: Context +) -> tuple[RetryTask | TaskState, TaskInstanceState, Exception]: + """ + Decide the outcome for an exception raised by one of the outcome handlers themselves. + + Handlers reach this by talking to the API server -- a missing Dag makes + ``_handle_trigger_dag_run`` raise ``AirflowRuntimeError`` -- or by serializing + user-supplied values, since ``_defer_task`` and ``_await_input_task`` run + ``serde_serialize`` over kwargs and it raises ``TypeError`` for anything it has no + serializer for. + + The handler chain's own classifications are preserved rather than routing everything + through the retry-count check, so an exception that means "do not retry" still means + that when it surfaces from a handler. + """ + log.exception("Task failed with exception") + if isinstance(exception, (AirflowFailException, AirflowSensorTimeout, AirflowTaskTerminated)): + return _terminal_failure(ti), TaskInstanceState.FAILED, exception + try: + msg, state = _handle_current_task_failed(ti, exception, log, context) + except Exception: + # The failure path itself failed. Re-entering it would just fail again and + # escape, losing the callbacks this whole function exists to preserve, so fail + # closed on a plain FAILED state instead. + log.exception("Could not determine terminal state, failing closed") + return _terminal_failure(ti), TaskInstanceState.FAILED, exception + return msg, state, exception + + +def _terminal_failure(ti: RuntimeTaskInstance) -> TaskState: + """Build a plain FAILED terminal message, bypassing any retry decision.""" + ti.end_date = datetime.now(tz=timezone.utc) + return TaskState( + state=TaskInstanceState.FAILED, + end_date=ti.end_date, + rendered_map_index=ti.rendered_map_index, + ) + + def _handle_current_task_failed( ti: RuntimeTaskInstance, exception: BaseException, @@ -1849,12 +1913,16 @@ def _finalize_task_failure( end_date = datetime.now(tz=timezone.utc) ti.end_date = end_date - # Record operator and task instance failed metrics - operator = ti.task.__class__.__name__ - stats_tags = ti.stats_tags + # Record operator and task instance failed metrics. One failure is one increment even + # if this runs twice, which happens when a first pass raised after counting -- see + # `_handle_handler_failure`. + if not ti._failure_metrics_emitted: + operator = ti.task.__class__.__name__ + stats_tags = ti.stats_tags - stats.incr("operator_failures", tags={**stats_tags, "operator_name": operator}) - stats.incr("ti_failures", tags=stats_tags) + stats.incr("operator_failures", tags={**stats_tags, "operator_name": operator}) + stats.incr("ti_failures", tags=stats_tags) + ti._failure_metrics_emitted = True if ti._ti_context_from_server and ti._ti_context_from_server.should_retry: retry_kwargs: dict[str, Any] = {"end_date": end_date} diff --git a/task-sdk/src/airflow/sdk/serde/serializers/pandas.py b/task-sdk/src/airflow/sdk/serde/serializers/pandas.py index a3562f5af4701..aaa68580abe9f 100644 --- a/task-sdk/src/airflow/sdk/serde/serializers/pandas.py +++ b/task-sdk/src/airflow/sdk/serde/serializers/pandas.py @@ -22,49 +22,15 @@ from airflow.sdk.module_loading import qualname # lazy loading for performance reasons -# -# ``pandas.core.frame.DataFrame`` is what a DataFrame qualifies as under pandas 2. Under -# pandas 3 the public classes moved to the ``pandas`` namespace, so the same object -# qualifies as ``pandas.DataFrame``. That name is registered too, but *only* so that the -# registry dispatches here and this module can refuse it with an actionable message -- -# without it the caller gets serde's generic "cannot serialize object of type" instead. -# See ``_reject_pandas_3`` below. +# pandas 3 moved the public classes to the `pandas` namespace, so a DataFrame is qualified as +# `pandas.DataFrame` there and as `pandas.core.frame.DataFrame` on pandas 2. Both are registered so +# that XComs serialized by either version stay readable. serializers = [ - "pandas.core.frame.DataFrame", "pandas.DataFrame", + "pandas.core.frame.DataFrame", ] deserializers = serializers -# First pandas major version whose DataFrame XComs this version of Airflow does not support. -_FIRST_UNSUPPORTED_PANDAS_MAJOR = 3 - - -def _reject_pandas_3(pd) -> None: - """ - Refuse a DataFrame XCom when the installed pandas is 3 or newer. - - pandas is an optional dependency, so the ``<3`` constraint Airflow declares cannot be - relied on: a deployment may install pandas 3 directly, or pull it in through another - package. This is the runtime half of that constraint. - - The failure is deliberate rather than best-effort. Under pandas 3 a DataFrame round - trips through parquet with different dtypes than under pandas 2 -- an ``object`` column - comes back as ``str`` and missing values as ``nan`` rather than ``None`` -- so silently - accepting the value would hand tasks data that differs from what was written, with no - signal. Failing on the write is the louder and more recoverable half of that. - """ - major = int(pd.__version__.split(".")[0]) - if major < _FIRST_UNSUPPORTED_PANDAS_MAJOR: - return - raise RuntimeError( - f"DataFrame XComs are not supported with pandas {pd.__version__}: this version of " - f"Airflow supports pandas < {_FIRST_UNSUPPORTED_PANDAS_MAJOR} for XCom serialization. " - "Pin pandas < 3 in the environment that runs your tasks, or avoid passing DataFrames " - "through XCom. Moving to pandas 3 is a deliberate migration -- its DataFrames read " - "back with different dtypes -- and Airflow will enable it in a future release." - ) - - if TYPE_CHECKING: import pandas as pd @@ -81,8 +47,6 @@ def serialize(o: object) -> tuple[U, str, int, bool]: if not isinstance(o, pd.DataFrame): return "", "", 0, False - _reject_pandas_3(pd) - # for now, we *always* serialize into in memory # until we have a generic backend that manages # sinks @@ -102,11 +66,6 @@ def deserialize(cls: type, version: int, data: object) -> pd.DataFrame: if cls is not pd.DataFrame: raise TypeError(f"do not know how to deserialize {qualname(cls)}") - # Reading is refused too, not just writing: a payload written under pandas 2 comes back - # with pandas 3 dtypes, so the task would silently receive different data than was - # pushed. An error is the safer outcome. - _reject_pandas_3(pd) - if not isinstance(data, str): raise TypeError(f"serialized {qualname(cls)} has wrong data type {type(data)}") diff --git a/task-sdk/tests/task_sdk/definitions/test_callback.py b/task-sdk/tests/task_sdk/definitions/test_callback.py index cc8562945bd59..cab8f3b28ad5f 100644 --- a/task-sdk/tests/task_sdk/definitions/test_callback.py +++ b/task-sdk/tests/task_sdk/definitions/test_callback.py @@ -16,11 +16,15 @@ # under the License. from __future__ import annotations +import importlib +import sys +from pathlib import Path from typing import cast import pytest from airflow.sdk._shared.module_loading import qualname +from airflow.sdk._shared.serialization import DATA from airflow.sdk.definitions.callback import AsyncCallback, Callback, SyncCallback from airflow.sdk.serde import deserialize, serialize @@ -39,6 +43,45 @@ def empty_sync_callback_for_deadline_tests(): TEST_CALLBACK_KWARGS = {"arg1": "value1"} UNIMPORTABLE_DOT_PATH = "valid.but.nonexistent.path" +# A module which leaves a trace when its body is executed, so that a test can tell whether +# it was imported. Written to a temporary directory by the `callback_module` fixture below. +CALLBACK_MODULE_SOURCE = ''' +from pathlib import Path + +Path(__file__).with_suffix(".imported").touch() + + +def sync_callback(): + """A callback which can be reached by dot path once this module is importable.""" + + +async def async_callback(): + """An awaitable callback which can be reached by dot path once this module is importable.""" +''' + + +@pytest.fixture +def callback_module(tmp_path, monkeypatch): + """ + Provide a factory which makes a named module importable for the duration of a test. + + The factory returns the module name and the path of a marker file which the module + creates when it is imported; the marker only exists if the module body has been run. + """ + monkeypatch.syspath_prepend(str(tmp_path)) + created: list[str] = [] + + def _create(module_name: str) -> tuple[str, Path]: + (tmp_path / f"{module_name}.py").write_text(CALLBACK_MODULE_SOURCE) + importlib.invalidate_caches() + created.append(module_name) + return module_name, tmp_path / f"{module_name}.imported" + + yield _create + + for module_name in created: + sys.modules.pop(module_name, None) + class TestCallback: @pytest.mark.parametrize( @@ -229,5 +272,89 @@ def test_serialize_deserialize(self): assert callback == deserialized +class TestCallbackPathHandling: + """Cover how a callback path is treated when it is supplied, versus read back from storage.""" + + def test_init_imports_the_module_named_by_the_path(self, callback_module): + """A Callback created from a dot path resolves it, so its author gets feedback on it.""" + module_name, marker = callback_module("callback_module_resolved_on_creation") + path = f"{module_name}.sync_callback" + + callback = SyncCallback(path) + + assert marker.exists(), "the module named by the path should have been imported" + assert callback.path == path + + def test_deserialize_does_not_import_the_module_named_by_the_stored_path(self, callback_module): + """Rebuilding a Callback keeps the stored path without resolving it again.""" + module_name, marker = callback_module("callback_module_not_resolved_on_deserialize") + path = f"{module_name}.sync_callback" + + callback = SyncCallback.deserialize({"path": path, "kwargs": {}, "executor": None}, 0) + + assert not marker.exists(), "the module named by the stored path should not have been imported" + assert module_name not in sys.modules + assert callback.path == path + assert type(callback.path) is str + + def test_deserialize_async_does_not_import_the_module_named_by_the_stored_path(self, callback_module): + module_name, marker = callback_module("async_callback_module_not_resolved_on_deserialize") + path = f"{module_name}.async_callback" + + callback = AsyncCallback.deserialize({"path": path, "kwargs": {}}, 0) + + assert not marker.exists(), "the module named by the stored path should not have been imported" + assert module_name not in sys.modules + assert callback.path == path + + def test_serde_deserialize_does_not_import_the_module_named_by_the_stored_path(self, callback_module): + """The same holds when the Callback is rebuilt through serde, as it is from stored data.""" + module_name, marker = callback_module("callback_module_not_resolved_through_serde") + path = f"{module_name}.sync_callback" + + serialized = serialize(SyncCallback(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS)) + serialized[DATA]["path"] = path + + deserialized = cast("Callback", deserialize(serialized)) + + assert not marker.exists(), "the module named by the stored path should not have been imported" + assert module_name not in sys.modules + assert deserialized.path == path + + @pytest.mark.parametrize( + "path", + [ + pytest.param("not a dot path", id="not_a_dot_path"), + pytest.param("", id="empty_string"), + pytest.param(None, id="none"), + pytest.param(42, id="not_a_string"), + ], + ) + def test_deserialize_rejects_a_path_which_is_not_a_dot_path(self, path): + """A stored value which is not shaped like a dot path is still rejected.""" + with pytest.raises(ImportError, match="doesn't look like a valid dot path."): + SyncCallback.deserialize({"path": path, "kwargs": {}, "executor": None}, 0) + + def test_deserialize_keeps_a_stored_path_which_no_longer_points_at_a_callable(self): + """ + The stored path is taken as it was stored. + + Unlike a path handed to the constructor, it is not checked against what it currently + resolves to; that check belongs to the moment the Callback is created. + """ + callback = SyncCallback.deserialize({"path": "os.path", "kwargs": {}, "executor": None}, 0) + + assert callback.path == "os.path" + + def test_deserialize_round_trip_keeps_kwargs_and_executor(self): + callback = SyncCallback(TEST_CALLBACK_PATH, kwargs=TEST_CALLBACK_KWARGS, executor="local") + + deserialized = cast("SyncCallback", deserialize(serialize(callback))) + + assert deserialized == callback + assert deserialized.kwargs == TEST_CALLBACK_KWARGS + assert deserialized.executor == "local" + + # While DeadlineReference lives in the SDK package, the unit tests to confirm it # works need database access so they live in the models/test_deadline.py module. diff --git a/task-sdk/tests/task_sdk/definitions/test_dag.py b/task-sdk/tests/task_sdk/definitions/test_dag.py index 9b76816886c76..07b8c8186c8e5 100644 --- a/task-sdk/tests/task_sdk/definitions/test_dag.py +++ b/task-sdk/tests/task_sdk/definitions/test_dag.py @@ -185,9 +185,7 @@ def test_non_empty_access_control_warns(self, role_access_control_entry: dict[st if len(role_access_control_entry) > 0 else role_access_control_entry ) - with pytest.warns( - RemovedInAirflow4Warning, match=re.escape("The airflow.security.permissions module is deprecated") - ): + with pytest.warns(RemovedInAirflow4Warning, match=re.escape("DAG.access_control is deprecated")): _ = DAG("should-warn-dag", access_control=access_control, schedule=None, start_date=DEFAULT_DATE) def test_params_not_passed_is_empty_dict(self): diff --git a/task-sdk/tests/task_sdk/execution_time/test_context.py b/task-sdk/tests/task_sdk/execution_time/test_context.py index f54b513093b0d..31009408ce4df 100644 --- a/task-sdk/tests/task_sdk/execution_time/test_context.py +++ b/task-sdk/tests/task_sdk/execution_time/test_context.py @@ -428,6 +428,36 @@ def test_var_json_masks_raw_string_and_dict_values(self, mock_mask_secret, mock_ # Second call: deserialized dict so internal sensitive fields like "password" get masked mock_mask_secret.assert_any_call({"password": "s3cr3t", "host": "db.example.com"}) + @mock.patch("airflow.sdk.execution_time.context.mask_secret") + def test_var_json_masks_list_values(self, mock_mask_secret, mock_supervisor_comms): + """A JSON list is handed to the masker whole, exactly as a dict is.""" + accessor = VariableAccessor(deserialize_json=True) + raw_json = '[{"password": "s3cr3t"}, {"password": "s3cr3t2"}]' + mock_supervisor_comms.send.return_value = VariableResult(key="db_configs", value=raw_json) + + val = accessor.db_configs + + assert val == [{"password": "s3cr3t"}, {"password": "s3cr3t2"}] + mock_mask_secret.assert_any_call(raw_json, "db_configs") + # under the variable's key; dicts inside are still masked by their own key names + mock_mask_secret.assert_any_call([{"password": "s3cr3t"}, {"password": "s3cr3t2"}], "db_configs") + + @pytest.mark.parametrize( + ("raw", "expected"), + [ + pytest.param("12345", 12345, id="int"), + pytest.param("true", True, id="bool"), + pytest.param("null", None, id="null"), + pytest.param("1.5", 1.5, id="float"), + ], + ) + def test_var_json_scalar_values_pass_through(self, raw, expected, mock_supervisor_comms): + """Handing a scalar to the masker is a no-op and must not change the value returned.""" + accessor = VariableAccessor(deserialize_json=True) + mock_supervisor_comms.send.return_value = VariableResult(key="some_number", value=raw) + + assert accessor.some_number == expected + @mock.patch("airflow.sdk.execution_time.context.mask_secret") def test_var_json_sensitive_key_masks_raw_json(self, mock_mask_secret, mock_supervisor_comms): """var.json. masks the entire raw JSON string because the variable key is sensitive.""" @@ -464,7 +494,10 @@ def test_var_json_list_value_does_not_over_mask(self, mock_mask_secret, mock_sup val = accessor.aws_regions assert val == ["us-east-1", "eu-west-1"] - mock_mask_secret.assert_called_once_with(raw_json, "aws_regions") + mock_mask_secret.assert_any_call(raw_json, "aws_regions") + mock_mask_secret.assert_any_call(["us-east-1", "eu-west-1"], "aws_regions") + # never anonymously -- that is what would mask the elements globally + assert mock.call(["us-east-1", "eu-west-1"]) not in mock_mask_secret.call_args_list @mock.patch("airflow.sdk.execution_time.context.mask_secret") def test_var_json_invalid_json_raises(self, mock_mask_secret): diff --git a/task-sdk/tests/task_sdk/execution_time/test_supervisor.py b/task-sdk/tests/task_sdk/execution_time/test_supervisor.py index f777b2d5a8a90..909f4942ef8ab 100644 --- a/task-sdk/tests/task_sdk/execution_time/test_supervisor.py +++ b/task-sdk/tests/task_sdk/execution_time/test_supervisor.py @@ -160,10 +160,12 @@ _ResponseFrame, ) from airflow.sdk.execution_time.supervisor import ( + SERVER_TERMINATED, ActivitySubprocess, InProcessSupervisorComms, InProcessTestSupervisor, ProcessTracker, + WatchedSubprocess, _make_process_nondumpable, _remote_logging_conn, in_process_api_server, @@ -1241,6 +1243,71 @@ def test_cleanup_sockets_after_delay(self, monkeypatch, mocker): proc.selector.close.assert_called_once() proc.stdin.close.assert_called_once() + def test_task_runner_starts_in_new_process_group(self, client_with_ti_start): + """Regression test for #65505: the task-runner child must be placed in + its own process group (PGID == its PID) so kill() can reach + subprocesses the task-runner spawns via os.killpg(); without it, a + venv/Popen child of the task-runner inherits the supervisor's process + group and killpg would signal the supervisor too (or miss the + grandchild entirely). + + The group must already exist when start() returns: the parent sets it + too (double setpgid), closing the race where kill() runs before the + child is first scheduled (e.g. task_instances.start() failing + synchronously in _on_child_started). + """ + + def subprocess_main(): + CommsDecoder()._get_response() + sleep(10) + + proc = ActivitySubprocess.start( + dag_rel_path=os.devnull, + bundle_info=FAKE_BUNDLE, + what=TaskInstance( + id=uuid7(), + task_id="b", + dag_id="c", + run_id="d", + try_number=1, + dag_version_id=uuid7(), + queue="default", + ), + client=client_with_ti_start, + target=subprocess_main, + ) + try: + child_pgid = os.getpgid(proc.pid) + assert child_pgid == proc.pid, ( + "Task-runner child must be its own process-group leader as soon " + f"as start() returns. Got pgid={child_pgid}, pid={proc.pid}." + ) + assert child_pgid != os.getpgid(0), ( + "Child's process group must differ from the supervisor's so " + "os.killpg() from kill() does not signal the supervisor itself." + ) + finally: + proc.kill(signal.SIGKILL, force=True) + proc.wait() + + def test_child_keeps_supervisor_process_group_by_default(self): + """Subprocess types that don't opt in to new_process_group (DAG + processor, triggerer, callbacks) must keep the supervisor's process + group: they install their own signal handlers and expect direct, + graceful signalling rather than group-wide delivery. + """ + + def subprocess_main(): + sleep(30) + + proc = WatchedSubprocess.start(id=uuid7(), target=subprocess_main) + try: + assert os.getpgid(proc.pid) == os.getpgid(0), ( + "Without new_process_group=True the child must stay in the supervisor's process group." + ) + finally: + proc.kill(signal.SIGKILL, force=True) + class TestWatchedSubprocessKill: @pytest.fixture @@ -1261,6 +1328,7 @@ def watched_subprocess(self, mocker, mock_process): stdin=mocker.Mock(), client=mocker.Mock(), process=mock_process, + new_process_group=True, ) # Mock the selector mock_selector = mocker.Mock(spec=selectors.DefaultSelector) @@ -1270,8 +1338,11 @@ def watched_subprocess(self, mocker, mock_process): proc.selector = mock_selector return proc - def test_kill_process_already_exited(self, watched_subprocess, mock_process): + def test_kill_process_already_exited(self, watched_subprocess, mock_process, mocker): """Test behavior when the process has already exited.""" + # When the process is gone, getpgid raises ProcessLookupError and the + # kill() path falls back to send_signal on the dead psutil.Process. + mocker.patch("os.getpgid", side_effect=ProcessLookupError) mock_process.wait.side_effect = psutil.NoSuchProcess(pid=1234) watched_subprocess.kill(signal.SIGINT, force=True) @@ -1279,16 +1350,99 @@ def test_kill_process_already_exited(self, watched_subprocess, mock_process): mock_process.wait.assert_called_once() assert watched_subprocess._exit_code == -1 - def test_kill_process_custom_signal(self, watched_subprocess, mock_process): - """Test that the process is killed with the correct signal.""" + def test_kill_process_custom_signal(self, watched_subprocess, mock_process, mocker): + """Test that the process is killed with the correct signal via killpg.""" + mock_getpgid = mocker.patch("os.getpgid", side_effect=lambda pid: 12345 if pid else 54321) + mock_killpg = mocker.patch("os.killpg") mock_process.wait.return_value = 0 signal_to_send = signal.SIGUSR1 watched_subprocess.kill(signal_to_send, force=False) - mock_process.send_signal.assert_called_once_with(signal_to_send) + assert mock_getpgid.call_args_list == [mocker.call(12345), mocker.call(0)] + mock_killpg.assert_called_once_with(12345, signal_to_send) + mock_process.send_signal.assert_not_called() mock_process.wait.assert_called_once_with(timeout=0) + def test_kill_signals_process_group(self, watched_subprocess, mock_process, mocker): + """Regression test for #65505: kill() must signal the whole process + group so subprocesses spawned by the task-runner (venv children, + Docker exec, bash shells) are also reached. + """ + mock_getpgid = mocker.patch("os.getpgid", side_effect=lambda pid: 12345 if pid else 54321) + mock_killpg = mocker.patch("os.killpg") + mock_process.wait.return_value = 0 + + watched_subprocess.kill(signal.SIGTERM, force=False) + + assert mock_getpgid.call_args_list == [mocker.call(12345), mocker.call(0)] + mock_killpg.assert_called_once_with(12345, signal.SIGTERM) + mock_process.send_signal.assert_not_called() + + def test_kill_does_not_signal_supervisors_own_process_group( + self, watched_subprocess, mock_process, mocker + ): + """If the child never made it into its own process group (setpgid + failed, or the child died and its PID's group resolves to ours), + os.killpg would signal the supervisor itself and every sibling in its + group -- and no exception would be raised for the fallback to catch. + kill() must detect the shared group and signal the child PID alone. + """ + mocker.patch("os.getpgid", return_value=54321) + mock_killpg = mocker.patch("os.killpg") + mock_process.wait.return_value = 0 + + watched_subprocess.kill(signal.SIGTERM, force=False) + + mock_killpg.assert_not_called() + mock_process.send_signal.assert_called_once_with(signal.SIGTERM) + + def test_kill_signals_pid_only_without_new_process_group(self, mocker, mock_process): + """Subprocess types that don't opt in to new_process_group (DAG + processor, triggerer, callbacks) must be signalled directly, never + via killpg. + """ + proc = ActivitySubprocess( + process_log=mocker.MagicMock(), + id=TI_ID, + pid=12345, + stdin=mocker.Mock(), + client=mocker.Mock(), + process=mock_process, + ) + mock_getpgid = mocker.patch("os.getpgid") + mock_killpg = mocker.patch("os.killpg") + mock_process.wait.return_value = 0 + + proc.kill(signal.SIGTERM, force=False) + + mock_getpgid.assert_not_called() + mock_killpg.assert_not_called() + mock_process.send_signal.assert_called_once_with(signal.SIGTERM) + + @pytest.mark.parametrize("failing_call", ["getpgid", "killpg"]) + @pytest.mark.parametrize("exc", [ProcessLookupError, PermissionError]) + def test_kill_falls_back_to_send_signal_when_group_signal_fails( + self, watched_subprocess, mock_process, mocker, failing_call, exc + ): + """If os.killpg or os.getpgid raises ProcessLookupError (group + vanished, e.g. task already reaped) or PermissionError, fall back to + signalling the task-runner PID directly via send_signal. + """ + if failing_call == "getpgid": + mocker.patch("os.getpgid", side_effect=exc) + mock_killpg = mocker.patch("os.killpg") + else: + mocker.patch("os.getpgid", side_effect=lambda pid: 12345 if pid else 54321) + mock_killpg = mocker.patch("os.killpg", side_effect=exc) + mock_process.wait.return_value = 0 + + watched_subprocess.kill(signal.SIGTERM, force=False) + + if failing_call == "getpgid": + mock_killpg.assert_not_called() + mock_process.send_signal.assert_called_once_with(signal.SIGTERM) + @pytest.mark.parametrize( ("signal_to_send", "exit_after"), [ @@ -3990,6 +4144,22 @@ def test_non_signal_exit_code_with_retry_goes_to_up_for_retry(self, mocker): assert mock_watched_subprocess.final_state == TaskInstanceState.UP_FOR_RETRY + def test_server_terminated_takes_precedence_over_retry(self, mocker): + """Test that a server-terminated task stays SERVER_TERMINATED even with retries enabled.""" + mock_watched_subprocess = ActivitySubprocess( + process_log=mocker.MagicMock(), + id=TI_ID, + pid=12345, + stdin=mocker.Mock(), + process=mocker.Mock(), + client=mocker.Mock(), + ) + mock_watched_subprocess._exit_code = 1 + mock_watched_subprocess._should_retry = True + mock_watched_subprocess._terminal_state = SERVER_TERMINATED + + assert mock_watched_subprocess.final_state == SERVER_TERMINATED + def test_non_signal_exit_code_without_retry_goes_to_failed(self, mocker): """Test that non-signal exit codes without retries enabled go to FAILED.""" mock_watched_subprocess = ActivitySubprocess( diff --git a/task-sdk/tests/task_sdk/execution_time/test_task_runner.py b/task-sdk/tests/task_sdk/execution_time/test_task_runner.py index cbd194e35ebf2..cc9fb77e08921 100644 --- a/task-sdk/tests/task_sdk/execution_time/test_task_runner.py +++ b/task-sdk/tests/task_sdk/execution_time/test_task_runner.py @@ -52,6 +52,7 @@ from airflow.listeners import hookimpl from airflow.providers.standard.operators.python import PythonOperator from airflow.providers.standard.operators.trigger_dagrun import TriggerDagRunOperator +from airflow.providers.standard.triggers.temporal import DateTimeTrigger from airflow.sdk import ( DAG, BaseOperator, @@ -79,7 +80,13 @@ from airflow.sdk.definitions._internal.types import NOTSET, SET_DURING_EXECUTION, is_arg_set from airflow.sdk.definitions.asset import Asset, AssetAlias, AssetUniqueKey, AssetUriRef, Dataset, Model from airflow.sdk.definitions.param import DagParam -from airflow.sdk.definitions.retry_policy import ExceptionRetryPolicy, RetryAction, RetryRule +from airflow.sdk.definitions.retry_policy import ( + ExceptionRetryPolicy, + RetryAction, + RetryDecision, + RetryPolicy, + RetryRule, +) from airflow.sdk.exceptions import ( AirflowException, AirflowFailException, @@ -133,6 +140,7 @@ PreviousTIResult, PrevSuccessfulDagRunResult, RescheduleTask, + RetryTask, SetAssetStateStoreByName, SetAssetStateStoreByUri, SetRenderedFields, @@ -980,6 +988,214 @@ def test_defer_task_queue_assignment( ) +@pytest.mark.parametrize( + ("should_retry", "expected_state"), + [ + (True, TaskInstanceState.UP_FOR_RETRY), + (False, TaskInstanceState.FAILED), + ], +) +def test_defer_with_unserializable_kwargs_honours_retries_and_callbacks( + should_retry, expected_state, create_runtime_ti, mock_supervisor_comms +): + """ + A task that defers with a non-serializable ``next_kwargs`` value must fail like any + other task, rather than taking the whole run down. + + ``_defer_task`` runs ``serde_serialize`` on the deferral kwargs, which raises + ``TypeError`` for anything it has no serializer for (a file handle, a client object, + a lambda). That raise happens inside ``run()``'s ``except TaskDeferred`` handler, so + before the fix it escaped ``run()`` without evaluating retries or running callbacks, + skipping the retry decision and callbacks even without involving the API server at + all. + """ + callbacks_run = [] + + class _DeferWithBadKwargs(BaseOperator): + def execute(self, context): + raise TaskDeferred( + trigger=DateTimeTrigger(moment=timezone.datetime(2024, 11, 22)), + method_name="next", + # A live handle is the realistic version of this mistake. + kwargs={"client": object()}, + ) + + task = _DeferWithBadKwargs( + task_id="defer_bad_kwargs", + on_failure_callback=lambda context: callbacks_run.append("failure"), + on_retry_callback=lambda context: callbacks_run.append("retry"), + ) + ti = create_runtime_ti( + dag_id="test_defer_with_unserializable_kwargs", + run_id="test_run", + task=task, + should_retry=should_retry, + ) + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + context = ti.get_template_context() + + state, _, error = run(ti, context, log) + + assert state == expected_state + assert isinstance(error, TypeError) + + context["exception"] = error + finalize(ti, state, context, log, error) + + assert callbacks_run == ["retry" if should_retry else "failure"] + + +def test_handler_failure_keeps_non_retryable_exceptions_non_retryable( + create_runtime_ti, mock_supervisor_comms +): + """ + A handler that raises a non-retryable exception must still fail without retrying. + + `AirflowFailException` means "do not retry" wherever it is raised. Routing handler + failures through `_handle_current_task_failed` would consult the retry count instead + and hand back UP_FOR_RETRY, so `_handle_handler_failure` keeps the main chain's + classification for these types. + """ + + class _FailingTrigger(DateTimeTrigger): + def serialize(self): + raise AirflowFailException("trigger cannot be serialized, do not retry") + + class _DeferWithFailingTrigger(BaseOperator): + def execute(self, context): + raise TaskDeferred( + trigger=_FailingTrigger(moment=timezone.datetime(2024, 11, 22)), + method_name="next", + ) + + task = _DeferWithFailingTrigger(task_id="defer_fail_exc") + # Retries are available; the exception type must still win. + ti = create_runtime_ti( + dag_id="test_handler_failure_non_retryable", + run_id="test_run", + task=task, + should_retry=True, + ) + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + + state, msg, error = run(ti, ti.get_template_context(), log) + + assert state == TaskInstanceState.FAILED + assert isinstance(msg, TaskState) + assert isinstance(error, AirflowFailException) + + +def test_handler_failure_lets_keyboard_interrupt_propagate(create_runtime_ti, mock_supervisor_comms): + """ + A ``KeyboardInterrupt`` inside a handler must reach ``main()``, not become a task failure. + + The supervisor's default termination signal is SIGINT, so swallowing it here would + convert an operator-initiated kill into an ordinary retry. + """ + + class _InterruptingTrigger(DateTimeTrigger): + def serialize(self): + raise KeyboardInterrupt + + class _DeferWithInterrupt(BaseOperator): + def execute(self, context): + raise TaskDeferred( + trigger=_InterruptingTrigger(moment=timezone.datetime(2024, 11, 22)), + method_name="next", + ) + + task = _DeferWithInterrupt(task_id="defer_interrupt") + ti = create_runtime_ti(dag_id="test_handler_interrupt", run_id="test_run", task=task) + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + + with pytest.raises(KeyboardInterrupt): + run(ti, ti.get_template_context(), log) + + +def test_handler_failure_fails_closed_when_failure_path_also_fails(create_runtime_ti, mock_supervisor_comms): + """ + If the failure path itself raises, `run()` must still return a terminal state. + + A `retry_policy` is user code and `RetryDecision` is an unvalidated dataclass, so a + policy handing back `retry_delay=30` (seconds, rather than a `timedelta`) reaches + `_finalize_task_failure`, which calls `.total_seconds()` on it. That `AttributeError` + is raised outside `_evaluate_retry_policy`'s own error handling, so it propagates out + of the failure path. Re-entering that path with its own exception would fail the same + way and escape `run()`, losing the callbacks this handling exists to preserve. + """ + + class _BadDelayPolicy(RetryPolicy): + def evaluate(self, exception, try_number, max_tries, context=None): + # Seconds as an int, not a timedelta. + return RetryDecision(action=RetryAction.RETRY, retry_delay=30) + + class _AlwaysFails(BaseOperator): + def execute(self, context): + raise RuntimeError("boom") + + task = _AlwaysFails(task_id="bad_delay_policy", retry_policy=_BadDelayPolicy()) + ti = create_runtime_ti( + dag_id="test_handler_double_fault", + run_id="test_run", + task=task, + should_retry=True, + ) + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + + state, msg, error = run(ti, ti.get_template_context(), log) + + assert state == TaskInstanceState.FAILED + assert isinstance(msg, TaskState) + assert msg.state == TaskInstanceState.FAILED + # The terminal state must actually reach the supervisor, not merely be returned. + assert any(call.kwargs.get("msg") is msg for call in mock_supervisor_comms.send.call_args_list) + # `error` is the exception that actually ended the run, so a callback sees the broken + # policy rather than a misleadingly clean task error. The task's own failure stays + # reachable on the implicit exception chain. + assert isinstance(error, AttributeError) + assert isinstance(error.__context__, RuntimeError) + + +def test_handler_failure_counts_the_failure_once(create_runtime_ti, mock_supervisor_comms): + """ + One failure is one increment, even when the failure path runs twice. + + The first pass through `_finalize_task_failure` records the counters before the broken + retry delay makes it raise, so counting again on the second pass would report two + failures for a single task run. + """ + + class _BadDelayPolicy(RetryPolicy): + def evaluate(self, exception, try_number, max_tries, context=None): + return RetryDecision(action=RetryAction.RETRY, retry_delay=30) + + class _AlwaysFails(BaseOperator): + def execute(self, context): + raise RuntimeError("boom") + + task = _AlwaysFails(task_id="count_once", retry_policy=_BadDelayPolicy()) + ti = create_runtime_ti( + dag_id="test_handler_failure_counts_once", + run_id="test_run", + task=task, + should_retry=True, + ) + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + stats_backend = mock.MagicMock(spec=StatsLogger) + + with mock.patch("airflow.sdk.execution_time.task_runner.stats", stats_backend): + run(ti, ti.get_template_context(), log) + + counted = [call.args[0] for call in stats_backend.incr.call_args_list if call.args] + assert counted.count("ti_failures") == 1 + assert counted.count("operator_failures") == 1 + + def test_run_downstream_skipped(mocked_parse, create_runtime_ti, mock_supervisor_comms, listener_manager): listener = TestTaskRunnerCallsListeners.CustomListener() listener_manager(listener) @@ -5153,10 +5369,10 @@ def test_handle_trigger_dag_run_conflict( mock_supervisor_comms.assert_has_calls(expected_calls) @time_machine.travel("2025-01-01 00:00:00", tick=False) - def test_handle_trigger_dag_run_reraises_original_error(self, create_runtime_ti, mock_supervisor_comms): + def test_handle_trigger_dag_run_surfaces_original_error(self, create_runtime_ti, mock_supervisor_comms): """ - When an ``except`` handler in ``run()`` raises before binding ``state``, - the original exception must propagate + When an ``except`` handler in ``run()`` raises, the original exception must be + surfaced as the task failure, not an ``UnboundLocalError`` on ``state``. """ from airflow.providers.standard.operators.trigger_dagrun import TriggerDagRunOperator @@ -5169,7 +5385,7 @@ class _TriggerSendError(Exception): trigger_run_id="test_run_id", ) ti = create_runtime_ti( - dag_id="test_handle_trigger_dag_run_reraises_original_error", + dag_id="test_handle_trigger_dag_run_surfaces_original_error", run_id="test_run", task=task, ) @@ -5183,11 +5399,85 @@ def _send(msg=None, **kwargs): mock_supervisor_comms.send.side_effect = _send - log = mock.MagicMock() + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + + state, _, error = run(ti, ti.get_template_context(), log) + + assert state == TaskInstanceState.FAILED + assert isinstance(error, _TriggerSendError) + + @pytest.mark.parametrize( + ("should_retry", "expected_state"), + [ + (True, TaskInstanceState.UP_FOR_RETRY), + (False, TaskInstanceState.FAILED), + ], + ) + @time_machine.travel("2025-01-01 00:00:00", tick=False) + def test_handle_trigger_dag_run_missing_dag_honours_retries_and_callbacks( + self, should_retry, expected_state, create_runtime_ti, mock_supervisor_comms + ): + """ + A 404 from the API server for a missing target Dag must be handled like any other + task failure: the retry decision is made and ``finalize()`` fires the callbacks. + + Regression test for https://github.com/apache/airflow/issues/70683 -- the + ``AirflowRuntimeError`` was raised from inside ``run()``'s + ``except DagRunTriggerException`` handler, so it escaped ``run()`` without + evaluating retries or running ``on_failure_callback`` / ``on_retry_callback``. + """ + callbacks_run = [] + + task = TriggerDagRunOperator( + task_id="test_task", + trigger_dag_id="this_dag_does_not_exist", + trigger_run_id="test_run_id", + on_failure_callback=lambda context: callbacks_run.append("failure"), + on_retry_callback=lambda context: callbacks_run.append("retry"), + ) + ti = create_runtime_ti( + dag_id="test_handle_trigger_dag_run_missing_dag", + run_id="test_run", + task=task, + should_retry=should_retry, + ) + + not_found = AirflowRuntimeError( + error=ErrorResponse( + error=ErrorType.API_SERVER_ERROR, + detail={ + "status_code": 404, + "message": "Client error message: Dag with dag_id: 'this_dag_does_not_exist' not found", + }, + ) + ) + + def _send(msg=None, **kwargs): + if isinstance(msg, TriggerDagRun): + raise not_found + return mock.DEFAULT + + mock_supervisor_comms.send.side_effect = _send + + log = mock.MagicMock(spec=structlog.typing.FilteringBoundLogger) + context = ti.get_template_context() + + state, msg, error = run(ti, context, log) + + assert state == expected_state + assert error is not_found + # The terminal message is what the server acts on, so assert the shape too: + # RetryTask drives UP_FOR_RETRY, TaskState(FAILED) ends the try. + if should_retry: + assert isinstance(msg, RetryTask) + else: + assert isinstance(msg, TaskState) + assert msg.state == TaskInstanceState.FAILED + + context["exception"] = error + finalize(ti, state, context, log, error) - # The original error must surface, not UnboundLocalError on ``state``. - with pytest.raises(_TriggerSendError): - run(ti, ti.get_template_context(), log) + assert callbacks_run == ["retry" if should_retry else "failure"] @pytest.mark.parametrize( ("allowed_states", "failed_states", "target_dr_state", "expected_task_state"), diff --git a/task-sdk/tests/task_sdk/serde/test_serializers.py b/task-sdk/tests/task_sdk/serde/test_serializers.py index 8ef21272194e8..30b9ea71d8d69 100644 --- a/task-sdk/tests/task_sdk/serde/test_serializers.py +++ b/task-sdk/tests/task_sdk/serde/test_serializers.py @@ -281,48 +281,45 @@ def test_pandas(self): d = deserialize(e) assert i.equals(d) - def test_pandas_serializers(self): - from airflow.sdk.serde.serializers.pandas import serialize - - assert serialize(123) == ("", "", 0, False) - - def test_pandas_3_dataframe_name_is_registered(self): - """The pandas 3 qualname must be registered, or the guard below is unreachable. - - Under pandas 3 a DataFrame qualifies as ``pandas.DataFrame``. serde dispatches on - that name, so if it is absent from the registry the caller gets the generic - "cannot serialize object of type" from serde and never reaches this module. + @pytest.mark.parametrize( + "classname", + ["pandas.DataFrame", "pandas.core.frame.DataFrame"], + ) + def test_pandas_deserializes_regardless_of_writer_qualname(self, classname): """ - from airflow.sdk.serde.serializers.pandas import deserializers, serializers - - assert "pandas.DataFrame" in serializers - assert "pandas.core.frame.DataFrame" in deserializers + A DataFrame XCom must deserialize under either pandas major's registry qualname. - @pytest.mark.parametrize("version", ["3.0.0", "3.0.5", "4.1.2"]) - def test_pandas_3_dataframe_xcom_is_refused(self, monkeypatch, version): - """pandas >= 3 must fail loudly on both write and read, not round trip silently.""" + The installed pandas only ever produces its own qualname, so this forges the tag to + force the lookup through the other registry entry. + """ import pandas as pd - from airflow.sdk.serde.serializers import pandas as pandas_serializer - - monkeypatch.setattr(pd, "__version__", version) - df = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]}) + i = pd.DataFrame(data={"col1": [1, 2], "col2": [3, 4]}) + e = serialize(i) + e[CLASSNAME] = classname + d = deserialize(e) + assert i.equals(d) - with pytest.raises(RuntimeError, match=rf"DataFrame XComs are not supported with pandas {version}"): - pandas_serializer.serialize(df) + def test_pandas_serializers(self): + from airflow.sdk.serde.serializers.pandas import serialize - with pytest.raises(RuntimeError, match=r"DataFrame XComs are not supported with pandas"): - pandas_serializer.deserialize(pd.DataFrame, 1, "") + assert serialize(123) == ("", "", 0, False) @pytest.mark.parametrize( ("klass", "version", "data", "msg"), [ - (pd.DataFrame, 999, "", r"serialized 999 of pandas.core.frame.DataFrame > 1"), # version too new + # pandas 3 qualifies the class as pandas.DataFrame, pandas 2 as pandas.core.frame.DataFrame + ( + pd.DataFrame, + 999, + "", + r"serialized 999 of pandas(\.core\.frame)?\.DataFrame > 1", + ), # version too new ( pd.DataFrame, 1, 123, - r"serialized pandas.core.frame.DataFrame has wrong data type .*", + r"serialized pandas(\.core\.frame)?\.DataFrame has wrong data type .*", ), # bad payload type (str, 1, "", r"do not know how to deserialize builtins.str"), # bad class ], diff --git a/uv.lock b/uv.lock index 0c7c14c1c52b0..76b7408a20fae 100644 --- a/uv.lock +++ b/uv.lock @@ -1032,7 +1032,6 @@ all = [ { name = "apache-airflow-providers-cohere" }, { name = "apache-airflow-providers-common-ai" }, { name = "apache-airflow-providers-common-compat" }, - { name = "apache-airflow-providers-common-dataquality" }, { name = "apache-airflow-providers-common-io" }, { name = "apache-airflow-providers-common-messaging" }, { name = "apache-airflow-providers-common-sql", extra = ["pandas", "polars"] }, @@ -1054,7 +1053,6 @@ all = [ { name = "apache-airflow-providers-grpc" }, { name = "apache-airflow-providers-hashicorp" }, { name = "apache-airflow-providers-http" }, - { name = "apache-airflow-providers-ibm-mq", marker = "platform_machine != 'aarch64' and platform_machine != 'arm64'" }, { name = "apache-airflow-providers-imap" }, { name = "apache-airflow-providers-influxdb" }, { name = "apache-airflow-providers-informatica" }, @@ -1217,9 +1215,6 @@ common-ai = [ common-compat = [ { name = "apache-airflow-providers-common-compat" }, ] -common-dataquality = [ - { name = "apache-airflow-providers-common-dataquality" }, -] common-io = [ { name = "apache-airflow-providers-common-io" }, ] @@ -1297,9 +1292,6 @@ hashicorp = [ http = [ { name = "apache-airflow-providers-http" }, ] -ibm-mq = [ - { name = "apache-airflow-providers-ibm-mq", marker = "platform_machine != 'aarch64' and platform_machine != 'arm64'" }, -] imap = [ { name = "apache-airflow-providers-imap" }, ] @@ -1632,8 +1624,6 @@ requires-dist = [ { name = "apache-airflow-providers-common-ai", marker = "extra == 'common-ai'", editable = "providers/common/ai" }, { name = "apache-airflow-providers-common-compat", marker = "extra == 'all'", editable = "providers/common/compat" }, { name = "apache-airflow-providers-common-compat", marker = "extra == 'common-compat'", editable = "providers/common/compat" }, - { name = "apache-airflow-providers-common-dataquality", marker = "extra == 'all'", editable = "providers/common/dataquality" }, - { name = "apache-airflow-providers-common-dataquality", marker = "extra == 'common-dataquality'", editable = "providers/common/dataquality" }, { name = "apache-airflow-providers-common-io", marker = "extra == 'all'", editable = "providers/common/io" }, { name = "apache-airflow-providers-common-io", marker = "extra == 'common-io'", editable = "providers/common/io" }, { name = "apache-airflow-providers-common-messaging", marker = "extra == 'all'", editable = "providers/common/messaging" }, @@ -1680,8 +1670,6 @@ requires-dist = [ { name = "apache-airflow-providers-hashicorp", marker = "extra == 'hashicorp'", editable = "providers/hashicorp" }, { name = "apache-airflow-providers-http", marker = "extra == 'all'", editable = "providers/http" }, { name = "apache-airflow-providers-http", marker = "extra == 'http'", editable = "providers/http" }, - { name = "apache-airflow-providers-ibm-mq", marker = "platform_machine != 'aarch64' and platform_machine != 'arm64' and extra == 'all'", editable = "providers/ibm/mq" }, - { name = "apache-airflow-providers-ibm-mq", marker = "platform_machine != 'aarch64' and platform_machine != 'arm64' and extra == 'ibm-mq'", editable = "providers/ibm/mq" }, { name = "apache-airflow-providers-imap", marker = "extra == 'all'", editable = "providers/imap" }, { name = "apache-airflow-providers-imap", marker = "extra == 'imap'", editable = "providers/imap" }, { name = "apache-airflow-providers-influxdb", marker = "extra == 'all'", editable = "providers/influxdb" }, @@ -1792,7 +1780,7 @@ requires-dist = [ { name = "sentry-sdk", marker = "extra == 'sentry'", specifier = ">=2.30.0" }, { name = "uv", marker = "extra == 'uv'", specifier = ">=0.11.29" }, ] -provides-extras = ["all-core", "async", "graphviz", "gunicorn", "kerberos", "memray", "otel", "statsd", "all-task-sdk", "airbyte", "akeyless", "alibaba", "amazon", "anthropic", "apache-cassandra", "apache-drill", "apache-druid", "apache-flink", "apache-hdfs", "apache-hive", "apache-iceberg", "apache-impala", "apache-kafka", "apache-kylin", "apache-livy", "apache-pig", "apache-pinot", "apache-spark", "apache-tinkerpop", "apprise", "arangodb", "asana", "atlassian-jira", "celery", "clickhousedb", "cloudant", "cncf-kubernetes", "cohere", "common-ai", "common-compat", "common-dataquality", "common-io", "common-messaging", "common-sql", "databricks", "datadog", "dbt-cloud", "dingding", "discord", "docker", "edge3", "elasticsearch", "exasol", "fab", "facebook", "ftp", "git", "github", "google", "grpc", "hashicorp", "http", "ibm-mq", "imap", "influxdb", "informatica", "jdbc", "jenkins", "keycloak", "microsoft-azure", "microsoft-mssql", "microsoft-psrp", "microsoft-winrm", "mongo", "mysql", "neo4j", "odbc", "openai", "openfaas", "openlineage", "opensearch", "opsgenie", "oracle", "pagerduty", "papermill", "pgvector", "pinecone", "postgres", "presto", "qdrant", "redis", "salesforce", "samba", "segment", "sendgrid", "sftp", "singularity", "slack", "smtp", "snowflake", "sqlite", "ssh", "standard", "tableau", "telegram", "teradata", "trino", "vertica", "vespa", "weaviate", "yandex", "ydb", "zendesk", "all", "aiobotocore", "apache-atlas", "apache-webhdfs", "amazon-aws-auth", "cloudpickle", "github-enterprise", "google-auth", "ldap", "pandas", "polars", "rabbitmq", "sentry", "s3fs", "uv"] +provides-extras = ["all-core", "async", "graphviz", "gunicorn", "kerberos", "memray", "otel", "statsd", "all-task-sdk", "airbyte", "akeyless", "alibaba", "amazon", "anthropic", "apache-cassandra", "apache-drill", "apache-druid", "apache-flink", "apache-hdfs", "apache-hive", "apache-iceberg", "apache-impala", "apache-kafka", "apache-kylin", "apache-livy", "apache-pig", "apache-pinot", "apache-spark", "apache-tinkerpop", "apprise", "arangodb", "asana", "atlassian-jira", "celery", "clickhousedb", "cloudant", "cncf-kubernetes", "cohere", "common-ai", "common-compat", "common-io", "common-messaging", "common-sql", "databricks", "datadog", "dbt-cloud", "dingding", "discord", "docker", "edge3", "elasticsearch", "exasol", "fab", "facebook", "ftp", "git", "github", "google", "grpc", "hashicorp", "http", "imap", "influxdb", "informatica", "jdbc", "jenkins", "keycloak", "microsoft-azure", "microsoft-mssql", "microsoft-psrp", "microsoft-winrm", "mongo", "mysql", "neo4j", "odbc", "openai", "openfaas", "openlineage", "opensearch", "opsgenie", "oracle", "pagerduty", "papermill", "pgvector", "pinecone", "postgres", "presto", "qdrant", "redis", "salesforce", "samba", "segment", "sendgrid", "sftp", "singularity", "slack", "smtp", "snowflake", "sqlite", "ssh", "standard", "tableau", "telegram", "teradata", "trino", "vertica", "vespa", "weaviate", "yandex", "ydb", "zendesk", "all", "aiobotocore", "apache-atlas", "apache-webhdfs", "amazon-aws-auth", "cloudpickle", "github-enterprise", "google-auth", "ldap", "pandas", "polars", "rabbitmq", "sentry", "s3fs", "uv"] [package.metadata.requires-dev] ci-image = [ @@ -1901,8 +1889,8 @@ requires-dist = [ { name = "boto3", specifier = ">=1.37.2" }, { name = "click", specifier = ">=8.3.0" }, { name = "filelock", specifier = ">=3.13.0" }, - { name = "flit", specifier = ">=3.12.0" }, - { name = "flit-core", specifier = ">=3.12.0" }, + { name = "flit", specifier = ">=3.12.0,<4" }, + { name = "flit-core", specifier = ">=3.12.0,<4" }, { name = "gitpython", specifier = ">=3.1.40" }, { name = "google-api-python-client", specifier = ">=2.142.0" }, { name = "google-auth-httplib2", specifier = ">=0.2.0" }, @@ -2676,7 +2664,7 @@ requires-dist = [ { name = "rich", specifier = ">=13.6.0" }, { name = "rich-click", marker = "extra == 'devscripts'", specifier = ">=1.9.7" }, { name = "rich-click", marker = "extra == 'docs'", specifier = ">=1.9.7" }, - { name = "ruff", specifier = "==0.15.22" }, + { name = "ruff", specifier = "==0.16.0" }, { name = "semver", specifier = ">=3.0.2" }, { name = "semver", marker = "extra == 'devscripts'", specifier = ">=3.0.2" }, { name = "sphinx", marker = "extra == 'docs'", specifier = ">=7" }, @@ -2712,7 +2700,7 @@ requires-dist = [ { name = "types-deprecated", marker = "extra == 'mypy'", specifier = ">=1.2.9.20240311" }, { name = "types-docutils", marker = "extra == 'mypy'", specifier = ">=0.21.0.20240704" }, { name = "types-markdown", marker = "extra == 'mypy'", specifier = ">=3.6.0.20240316" }, - { name = "types-paramiko", marker = "extra == 'mypy'", specifier = ">=4.0.0.20260402,<5.0.0" }, + { name = "types-paramiko", marker = "extra == 'mypy'", specifier = ">=4.0.0.20260402" }, { name = "types-protobuf", marker = "extra == 'mypy'", specifier = ">=5.26.0.20240422" }, { name = "types-pymysql", marker = "extra == 'mypy'", specifier = ">=1.1.0.20240425" }, { name = "types-python-dateutil", marker = "extra == 'mypy'", specifier = ">=2.9.0.20240316" }, @@ -3180,9 +3168,9 @@ requires-dist = [ { name = "jsonpath-ng", specifier = ">=1.5.3" }, { name = "lxml", marker = "python_full_version < '3.13' and extra == 'python3-saml'", specifier = ">=6.0.0" }, { name = "marshmallow", specifier = ">=3" }, - { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3" }, { name = "pyathena", specifier = ">=3.10.0" }, { name = "python3-saml", marker = "python_full_version < '3.13' and extra == 'python3-saml'", specifier = ">=1.16.0" }, { name = "redshift-connector", specifier = ">=2.1.3" }, @@ -3488,9 +3476,9 @@ requires-dist = [ { name = "fastavro", marker = "python_full_version >= '3.14'", specifier = ">=1.12.1" }, { name = "hdfs", extras = ["avro", "dataframe", "kerberos"], marker = "python_full_version < '3.12'", specifier = ">=2.5.4" }, { name = "hdfs", extras = ["avro", "dataframe", "kerberos"], marker = "python_full_version >= '3.12'", specifier = ">=2.7.3" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, ] [package.metadata.requires-dev] @@ -3578,9 +3566,9 @@ requires-dist = [ { name = "hmsclient", specifier = ">=0.1.0" }, { name = "jmespath", specifier = ">=0.7.0" }, { name = "kerberos", marker = "sys_platform != 'win32' and extra == 'gssapi'", specifier = ">=1.3.0" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "pyhive", extras = ["hive-pure-sasl"], specifier = ">=0.7.0" }, { name = "sqlalchemy", marker = "extra == 'sqlalchemy'", specifier = ">=1.4.54" }, { name = "winkerberos", marker = "sys_platform == 'win32' and extra == 'gssapi'", specifier = ">=0.7.0" }, @@ -4223,7 +4211,7 @@ docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "d [[package]] name = "apache-airflow-providers-clickhousedb" -version = "1.0.1" +version = "1.0.0" source = { editable = "providers/clickhousedb" } dependencies = [ { name = "apache-airflow" }, @@ -4763,9 +4751,9 @@ requires-dist = [ { name = "datafusion", marker = "extra == 'datafusion'", specifier = ">=50.0.0,<52.0.0" }, { name = "methodtools", specifier = ">=0.4.7" }, { name = "more-itertools", specifier = ">=9.0.0" }, - { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3,<3" }, - { name = "pandas", extras = ["sql-other"], marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2,<3" }, + { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3" }, + { name = "pandas", extras = ["sql-other"], marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2" }, { name = "polars", marker = "extra == 'polars'", specifier = ">=1.26.0" }, { name = "pyiceberg-core", marker = "extra == 'pyiceberg-core'", specifier = ">=0.8.0" }, { name = "sqlalchemy", marker = "extra == 'sqlalchemy'", specifier = ">=1.4.54" }, @@ -4877,9 +4865,9 @@ requires-dist = [ { name = "fastavro", marker = "python_full_version >= '3.14' and extra == 'avro'", specifier = ">=1.12.1" }, { name = "fastavro", marker = "python_full_version < '3.14' and extra == 'avro'", specifier = ">=1.9.0" }, { name = "mergedeep", specifier = ">=1.3.4" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "pyarrow", marker = "python_full_version < '3.13'", specifier = ">=16.1.0" }, { name = "pyarrow", marker = "python_full_version == '3.13.*'", specifier = ">=18.0.0" }, { name = "pyarrow", marker = "python_full_version >= '3.14'", specifier = ">=22.0.0" }, @@ -5241,9 +5229,9 @@ requires-dist = [ { name = "apache-airflow", editable = "." }, { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, { name = "apache-airflow-providers-common-sql", editable = "providers/common/sql" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "pyexasol", specifier = ">=0.26.0,<2" }, { name = "sqlalchemy", marker = "extra == 'sqlalchemy'", specifier = ">=1.4.54" }, ] @@ -5520,7 +5508,7 @@ dependencies = [ { name = "google-api-python-client" }, { name = "google-auth" }, { name = "google-auth-httplib2" }, - { name = "google-cloud-aiplatform", extra = ["evaluation"] }, + { name = "google-cloud-aiplatform" }, { name = "google-cloud-alloydb" }, { name = "google-cloud-automl" }, { name = "google-cloud-batch" }, @@ -5564,6 +5552,9 @@ dependencies = [ { name = "grpcio-gcp" }, { name = "httpx" }, { name = "immutabledict" }, + { name = "jsonschema" }, + { name = "litellm", version = "1.85.7", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14'" }, + { name = "litellm", version = "1.95.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" }, { name = "looker-sdk" }, { name = "pandas" }, { name = "pandas-gbq" }, @@ -5571,10 +5562,15 @@ dependencies = [ { name = "pyarrow" }, { name = "pydantic", marker = "python_full_version >= '3.14'" }, { name = "python-slugify" }, + { name = "pyyaml" }, { name = "ray", extra = ["default"], marker = "python_full_version < '3.15'" }, + { name = "ruamel-yaml" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scikit-learn", version = "1.9.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, { name = "sqlalchemy-bigquery" }, { name = "sqlalchemy-spanner" }, { name = "tenacity" }, + { name = "tqdm" }, { name = "types-protobuf" }, ] @@ -5713,7 +5709,7 @@ requires-dist = [ { name = "google-api-python-client", specifier = ">=2.0.2" }, { name = "google-auth", specifier = ">=2.29.0" }, { name = "google-auth-httplib2", specifier = ">=0.0.1" }, - { name = "google-cloud-aiplatform", extras = ["evaluation"], specifier = ">=1.155.0" }, + { name = "google-cloud-aiplatform", specifier = ">=1.155.0" }, { name = "google-cloud-alloydb", specifier = ">=0.4.0" }, { name = "google-cloud-automl", specifier = ">=2.12.0" }, { name = "google-cloud-batch", specifier = ">=0.13.0" }, @@ -5758,10 +5754,13 @@ requires-dist = [ { name = "grpcio-gcp", specifier = ">=0.2.2" }, { name = "httpx", specifier = ">=0.25.0" }, { name = "immutabledict", specifier = ">=4.2.0" }, + { name = "jsonschema", specifier = ">=4.19.1" }, + { name = "litellm", marker = "python_full_version < '3.14'", specifier = ">=1.83.7,<1.86.0" }, + { name = "litellm", marker = "python_full_version >= '3.14'", specifier = ">=1.93.0" }, { name = "looker-sdk", specifier = ">=22.4.0,!=24.18.0" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "pandas-gbq", specifier = ">=0.7.0" }, { name = "plyvel", marker = "python_full_version < '3.13' and extra == 'leveldb'", specifier = ">=1.5.1" }, { name = "proto-plus", specifier = ">=1.26.0" }, @@ -5769,12 +5768,16 @@ requires-dist = [ { name = "pyarrow", marker = "python_full_version >= '3.14'", specifier = ">=22.0.0" }, { name = "pydantic", marker = "python_full_version >= '3.14'", specifier = ">=2.13.1" }, { name = "python-slugify", specifier = ">=7.0.0" }, + { name = "pyyaml", specifier = ">=6.0.3" }, { name = "ray", extras = ["default"], marker = "python_full_version < '3.13'", specifier = ">=2.42.0" }, { name = "ray", extras = ["default"], marker = "python_full_version == '3.13.*'", specifier = ">=2.49.0" }, { name = "ray", extras = ["default"], marker = "python_full_version == '3.14.*'", specifier = ">=2.55.0" }, + { name = "ruamel-yaml", specifier = ">=0.19.0" }, + { name = "scikit-learn", specifier = ">=1.6.0" }, { name = "sqlalchemy-bigquery", specifier = ">=1.2.1" }, { name = "sqlalchemy-spanner", specifier = ">=1.6.2" }, { name = "tenacity", specifier = ">=8.3.0" }, + { name = "tqdm", specifier = ">=4.23.0" }, { name = "types-protobuf", specifier = ">=5.27.0,!=5.29.1.20250402" }, ] provides-extras = ["cncf-kubernetes", "fab", "leveldb", "oracle", "facebook", "amazon", "apache-cassandra", "microsoft-azure", "microsoft-mssql", "mongo", "mysql", "openlineage", "postgres", "presto", "salesforce", "sftp", "ssh", "trino", "http", "standard", "common-messaging"] @@ -6873,7 +6876,7 @@ docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "d [[package]] name = "apache-airflow-providers-opensearch" -version = "1.11.2" +version = "1.12.0" source = { editable = "providers/opensearch" } dependencies = [ { name = "apache-airflow" }, @@ -7085,9 +7088,9 @@ requires-dist = [ { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, { name = "ipykernel", specifier = ">=6.29.4" }, { name = "nbconvert", specifier = ">=7.16.1" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "papermill", extras = ["all"], specifier = ">=2.6.0" }, { name = "scrapbook", extras = ["all"], specifier = ">=0.5.0" }, ] @@ -7253,9 +7256,9 @@ requires-dist = [ { name = "apache-airflow-providers-openlineage", marker = "extra == 'openlineage'", editable = "providers/openlineage" }, { name = "asyncpg", specifier = ">=0.30.0" }, { name = "asyncpg", marker = "extra == 'asyncpg'", specifier = ">=0.30.0" }, - { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version == '3.13.*' and extra == 'pandas'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version < '3.13' and extra == 'pandas'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version >= '3.14' and extra == 'pandas'", specifier = ">=2.3.3" }, { name = "polars", marker = "extra == 'polars'", specifier = ">=1.26.0" }, { name = "psycopg", extras = ["binary"], marker = "python_full_version < '3.14'", specifier = ">=3.2.9" }, { name = "psycopg", extras = ["binary"], marker = "python_full_version >= '3.14'", specifier = ">=3.3.3" }, @@ -7326,9 +7329,9 @@ requires-dist = [ { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, { name = "apache-airflow-providers-common-sql", editable = "providers/common/sql" }, { name = "apache-airflow-providers-google", marker = "extra == 'google'", editable = "providers/google" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "presto-python-client", specifier = ">=0.8.4" }, { name = "psycopg2-binary", marker = "python_full_version < '3.13'", specifier = ">=2.9.9" }, { name = "psycopg2-binary", marker = "python_full_version >= '3.13'", specifier = ">=2.9.10" }, @@ -7459,9 +7462,9 @@ docs = [ requires-dist = [ { name = "apache-airflow", editable = "." }, { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "simple-salesforce", specifier = ">=1.0.0" }, ] @@ -7488,6 +7491,10 @@ dependencies = [ google = [ { name = "apache-airflow-providers-google" }, ] +kerberos = [ + { name = "krb5" }, + { name = "smbprotocol", extra = ["kerberos"] }, +] [package.dev-dependencies] dev = [ @@ -7506,9 +7513,11 @@ requires-dist = [ { name = "apache-airflow", editable = "." }, { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, { name = "apache-airflow-providers-google", marker = "extra == 'google'", editable = "providers/google" }, + { name = "krb5", marker = "extra == 'kerberos'" }, { name = "smbprotocol", specifier = ">=1.5.0" }, + { name = "smbprotocol", extras = ["kerberos"], marker = "extra == 'kerberos'", specifier = ">=1.5.0" }, ] -provides-extras = ["google"] +provides-extras = ["google", "kerberos"] [package.metadata.requires-dev] dev = [ @@ -7788,6 +7797,7 @@ dependencies = [ { name = "apache-airflow-providers-common-sql" }, { name = "pandas" }, { name = "pyarrow" }, + { name = "pyopenssl" }, { name = "setuptools" }, { name = "snowflake-connector-python" }, { name = "snowflake-snowpark-python", marker = "python_full_version < '3.14'" }, @@ -7824,12 +7834,13 @@ requires-dist = [ { name = "apache-airflow-providers-common-sql", editable = "providers/common/sql" }, { name = "apache-airflow-providers-microsoft-azure", marker = "extra == 'microsoft-azure'", editable = "providers/microsoft/azure" }, { name = "apache-airflow-providers-openlineage", marker = "extra == 'openlineage'", editable = "providers/openlineage" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "pyarrow", marker = "python_full_version < '3.13'", specifier = ">=16.1.0" }, { name = "pyarrow", marker = "python_full_version == '3.13.*'", specifier = ">=18.0.0" }, { name = "pyarrow", marker = "python_full_version >= '3.14'", specifier = ">=22.0.0" }, + { name = "pyopenssl", specifier = ">=22.1.0" }, { name = "setuptools", specifier = ">=80.0.0,<9999" }, { name = "snowflake-connector-python", specifier = ">=3.17.0" }, { name = "snowflake-snowpark-python", marker = "python_full_version < '3.12'", specifier = ">=1.17.0,<9999" }, @@ -8159,9 +8170,9 @@ requires-dist = [ { name = "apache-airflow-providers-common-sql", editable = "providers/common/sql" }, { name = "apache-airflow-providers-google", marker = "extra == 'google'", editable = "providers/google" }, { name = "apache-airflow-providers-openlineage", marker = "extra == 'openlineage'", editable = "providers/openlineage" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "trino", specifier = ">=0.319.0" }, ] provides-extras = ["google", "openlineage"] @@ -8297,9 +8308,9 @@ requires-dist = [ { name = "apache-airflow", editable = "." }, { name = "apache-airflow-providers-common-compat", editable = "providers/common/compat" }, { name = "httpx", specifier = ">=0.25.0" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, { name = "weaviate-client", specifier = ">=4.16.0,!=4.16.7" }, ] @@ -9070,9 +9081,9 @@ dev = [ { name = "apache-airflow-devel-common", editable = "devel-common" }, { name = "apache-airflow-providers-common-sql", editable = "providers/common/sql" }, { name = "apache-airflow-providers-standard", editable = "providers/standard" }, - { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2,<3" }, - { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3,<3" }, - { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3,<3" }, + { name = "pandas", marker = "python_full_version < '3.13'", specifier = ">=2.1.2" }, + { name = "pandas", marker = "python_full_version == '3.13.*'", specifier = ">=2.2.3" }, + { name = "pandas", marker = "python_full_version >= '3.14'", specifier = ">=2.3.3" }, ] docs = [{ name = "apache-airflow-devel-common", extras = ["docs"], editable = "devel-common" }] mypy = [{ name = "apache-airflow-devel-common", extras = ["mypy"], editable = "devel-common" }] @@ -12719,18 +12730,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/10/21/697400f2dd96377f0d0d41dcad67d6f8dda2f78232e0f9b1086ba1fef855/google_cloud_aiplatform-1.161.0-py2.py3-none-any.whl", hash = "sha256:e24c77461f046cdf846520ab943b7deccdbcbac98a143f24cd4dbab23a68950b", size = 9391446, upload-time = "2026-07-15T03:48:16.172Z" }, ] -[package.optional-dependencies] -evaluation = [ - { name = "jsonschema" }, - { name = "litellm" }, - { name = "pandas" }, - { name = "pyyaml" }, - { name = "ruamel-yaml" }, - { name = "scikit-learn", version = "1.5.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "scikit-learn", version = "1.9.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, - { name = "tqdm" }, -] - [[package]] name = "google-cloud-alloydb" version = "0.11.0" @@ -15655,6 +15654,16 @@ wheels = [ name = "litellm" version = "1.85.7" source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version == '3.13.*' and platform_machine == 'arm64' and sys_platform == 'darwin'", + "(python_full_version == '3.13.*' and platform_machine != 'arm64') or (python_full_version == '3.13.*' and sys_platform != 'darwin')", + "python_full_version == '3.12.*' and platform_machine == 'arm64' and sys_platform == 'darwin'", + "(python_full_version == '3.12.*' and platform_machine != 'arm64') or (python_full_version == '3.12.*' and sys_platform != 'darwin')", + "python_full_version == '3.11.*' and platform_machine == 'arm64' and sys_platform == 'darwin'", + "(python_full_version == '3.11.*' and platform_machine != 'arm64') or (python_full_version == '3.11.*' and sys_platform != 'darwin')", + "python_full_version < '3.11' and platform_machine == 'arm64' and sys_platform == 'darwin'", + "(python_full_version < '3.11' and platform_machine != 'arm64') or (python_full_version < '3.11' and sys_platform != 'darwin')", +] dependencies = [ { name = "aiohttp" }, { name = "click" }, @@ -15674,6 +15683,49 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bd/3b/baf1f8736c572f308eea14100372c39b3f70ee540e8cda537edd594628fc/litellm-1.85.7-py3-none-any.whl", hash = "sha256:4e07ec1c22aa81897daed83d7e98eb05187a82827918c240e2952384780f9c62", size = 16996595, upload-time = "2026-06-24T04:54:51.974Z" }, ] +[[package]] +name = "litellm" +version = "1.95.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.15' and sys_platform == 'win32'", + "python_full_version >= '3.15' and sys_platform == 'emscripten'", + "python_full_version >= '3.15' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version == '3.14.*'", +] +dependencies = [ + { name = "aiohttp" }, + { name = "click" }, + { name = "fastuuid" }, + { name = "httpx" }, + { name = "importlib-metadata" }, + { name = "jinja2" }, + { name = "jsonschema" }, + { name = "openai" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "tiktoken" }, + { name = "tokenizers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/96/8cdfb9aaf584b57af35a0423c111a1c1264a78b548cebbb5ed96defacdab/litellm-1.95.0.tar.gz", hash = "sha256:0ef126d52c7a559f8353e50d60fd0d5e7e6c8767ad54df25ddaf79b9edca1afc", size = 17513577, upload-time = "2026-08-02T02:52:49.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/50/e3/4ba824200c235c56bc87759f05bde5c53223f8036282026b2c731af7eb9a/litellm-1.95.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:0106b3564b60d00cb5b2810824ebf5071f59c9f9262318884d9c6f040aa2c435", size = 26426637, upload-time = "2026-08-02T02:51:53.675Z" }, + { url = "https://files.pythonhosted.org/packages/22/09/38d6a4a53be2cca1e4efa05a4233b791efe82ba92a10088630235132933f/litellm-1.95.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:053cea1a584bf92d5d44b422f99fa03715bf7c346c8fe080c29bf0c6bd71eddc", size = 26304372, upload-time = "2026-08-02T02:51:58.15Z" }, + { url = "https://files.pythonhosted.org/packages/5c/dc/3c4c46343188fda4aa6b7294aff9969beb58dae400130a0f09c774d23f61/litellm-1.95.0-cp310-cp310-win_amd64.whl", hash = "sha256:667cc7cc58e05a9f9c4bf4588cd4ff5c785fd265060acdfb9147332b75b73ce9", size = 24920517, upload-time = "2026-08-02T02:52:01.547Z" }, + { url = "https://files.pythonhosted.org/packages/83/c1/470070205a3b36c4d99d95046102468165a55bfd675f32c552566085e746/litellm-1.95.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:4bafa3494d503a3c6c1f2eeb785a2af364d85463c71a92cabaaa761c9227d62a", size = 26425608, upload-time = "2026-08-02T02:52:05.277Z" }, + { url = "https://files.pythonhosted.org/packages/ec/c1/54c79849f4b60aab55772d69099557e49c86c6a43a863d65c1bc985246b3/litellm-1.95.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:4c2a06d2263a07a29228cd3af2b594cf86cd3a4182a7324c517cba88a9415969", size = 26303794, upload-time = "2026-08-02T02:52:08.675Z" }, + { url = "https://files.pythonhosted.org/packages/c5/52/31f2f50063e619b782c74df3ea8835270cce40f94dadaed350cf2f499aa0/litellm-1.95.0-cp311-cp311-win_amd64.whl", hash = "sha256:aac37bb6d2be191bafc0ce590ed06d21dd7e5a37599ffc1af1071899f6c74b73", size = 24921269, upload-time = "2026-08-02T02:52:12.281Z" }, + { url = "https://files.pythonhosted.org/packages/33/d0/ad0272853cc450f8bb4a40a93d206e767e18ac2ff3f91870374e1d9fc090/litellm-1.95.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:cb667f84f08520f32b076e03c7a3fa51bf3f7e8b641dade34ab046bf00314d6b", size = 26421359, upload-time = "2026-08-02T02:52:16.048Z" }, + { url = "https://files.pythonhosted.org/packages/02/c1/4301aa8ef6d2fb0e4a2b8dec973d7c4499f680b26d2e1b77643864235a4b/litellm-1.95.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:1bdf7153557cc0851fa9477b137fde476c56d5de92a5778ecfc6c3a75439a4e1", size = 26300401, upload-time = "2026-08-02T02:52:19.501Z" }, + { url = "https://files.pythonhosted.org/packages/7c/3d/6cd087bd541f18d924f17bd8e1bb68a7f9d73d03274c5339f9de563bc992/litellm-1.95.0-cp312-cp312-win_amd64.whl", hash = "sha256:62cc5d834e8223dbd16c9ad0b46c73354b6d67cc7fa0eba2764ce65b3b8c474f", size = 24917446, upload-time = "2026-08-02T02:52:23.119Z" }, + { url = "https://files.pythonhosted.org/packages/fd/47/719785f65b01779cf7568c329430c93b2e7832498deb3deec53bdd106f8d/litellm-1.95.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:9d80a9adc506bfce48145621d6649e3fd428407811eb00211bcce33344054701", size = 26422310, upload-time = "2026-08-02T02:52:26.626Z" }, + { url = "https://files.pythonhosted.org/packages/55/48/06447e1125d7ae31bd24d34d2af2833b15aed79dc5f7e8b45862c1d06af8/litellm-1.95.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:cf014ff515825ad49937b4cdf95616270789311db7841d16702e0a5b1ac5b067", size = 26300935, upload-time = "2026-08-02T02:52:30.032Z" }, + { url = "https://files.pythonhosted.org/packages/45/6f/388f85ebcb4e239dc738cab99307051d5a8a69d907023b0dbb200ee95226/litellm-1.95.0-cp313-cp313-win_amd64.whl", hash = "sha256:c73df441153e585832d4e90e3717d17ae888b269daa71d723336369e81ef884b", size = 24917355, upload-time = "2026-08-02T02:52:33.593Z" }, + { url = "https://files.pythonhosted.org/packages/bb/f6/5b2e944d5c8cea5164e10a28220d9af598792a9c3f25605bcf14946323ef/litellm-1.95.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:c8e712f95764a9a730f3aec9dff8be916973423411ca12a79a35c8bea3f7d5a4", size = 26423318, upload-time = "2026-08-02T02:52:37.629Z" }, + { url = "https://files.pythonhosted.org/packages/ca/f7/d7450f5d3d7b566900067a21bdfafb5cee62bc0bc73b190e8598df1f2e67/litellm-1.95.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:071a63c0e1d949bf7ed5e5c73843cb042a6e00324d3f214b13f8c6b90da6822d", size = 26299965, upload-time = "2026-08-02T02:52:41.577Z" }, + { url = "https://files.pythonhosted.org/packages/c9/d4/c2b7d8b239021c7737a228ab5ffe4f40e4a596ce4629ff3c8899bfd40904/litellm-1.95.0-cp314-cp314-win_amd64.whl", hash = "sha256:c3684dcf16aefe98bd6f11586d60a9a92bb1bf7e85468a6a4ac28a65532fab3e", size = 24920863, upload-time = "2026-08-02T02:52:45.663Z" }, +] + [[package]] name = "llama-index-core" version = "0.14.23" @@ -21523,27 +21575,27 @@ wheels = [ [[package]] name = "ruff" -version = "0.15.22" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/3a/06/ae069393fc66e8ff33036d4b368003833bf6e88ccf182e17e7a2f1c754fd/ruff-0.15.22.tar.gz", hash = "sha256:3f15175b1fb580126f58285a5dae6b2ea89000136d980c64499211f116b54809", size = 4785063, upload-time = "2026-07-16T15:14:13.244Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/23/18/ee54b7ae1e121be7a28ea6da4b67564ebb0530e183a54415ab7e3bcd2c4e/ruff-0.15.22-py3-none-linux_armv6l.whl", hash = "sha256:44423e73493737f5e7c5b41d475483898ff37afcdae38bc3da5085e29af1c2d8", size = 10781258, upload-time = "2026-07-16T15:13:19.452Z" }, - { url = "https://files.pythonhosted.org/packages/2f/d2/2520cb14761ddbeaf57642a76942fc36adcbdbe53b4532241995f6fc485c/ruff-0.15.22-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:b82c6482946e9eda7ff2e091d25b8bad3f718684e1916d41bd56873cee05b697", size = 10999477, upload-time = "2026-07-16T15:13:23.318Z" }, - { url = "https://files.pythonhosted.org/packages/c9/10/74e53572aa758dfaa678c2a2646b5c5515d884b7ca56be4d2ce03ca4b560/ruff-0.15.22-py3-none-macosx_11_0_arm64.whl", hash = "sha256:11c1c715af53a09f714e011106bffc419751ec8232fcb5da42173284ea3fec6f", size = 10466716, upload-time = "2026-07-16T15:13:26.162Z" }, - { url = "https://files.pythonhosted.org/packages/1e/cc/44eaaf0844e028182f2d0a8f2190d0f359159aed0a9e5ab861d892f1ae2a/ruff-0.15.22-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:742a29cf29bddb7c8327895d6a10e0e6c5b38a96dd407af9b5d0857f809c0576", size = 10892644, upload-time = "2026-07-16T15:13:29.229Z" }, - { url = "https://files.pythonhosted.org/packages/9f/21/8edf559014d2b0f82beea19cfb713993ad802ccda16868769979c6090a84/ruff-0.15.22-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72af58b951b0ae395935ae79763dc349bc0eb706319d28f7a33ad2cfb3cfc178", size = 10576719, upload-time = "2026-07-16T15:13:32.35Z" }, - { url = "https://files.pythonhosted.org/packages/bf/1e/3a13abd392a3b50b62e5938a831f9ab6e588358cacad5c18545b716d2182/ruff-0.15.22-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:62d425005c1835eb24e2ee4161cb90e8db263415f4a71c8c72c33abaa6c0c224", size = 11376494, upload-time = "2026-07-16T15:13:35.958Z" }, - { url = "https://files.pythonhosted.org/packages/bf/3e/422d3d95bcf04dd78e1aeac22184d4f9a8fb2c01865d39d44618484a0317/ruff-0.15.22-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e8b9b3f8779a4f08c969defc3c8c35abffaa757e601ed5ae66d6d1db6519969a", size = 12208370, upload-time = "2026-07-16T15:13:39.185Z" }, - { url = "https://files.pythonhosted.org/packages/1e/91/5d065a0e0a02bf4813f5119ad278462eed081d2b832eb7c021ade0ec9e65/ruff-0.15.22-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1e0dd1b2e4d3d585f897a0d137cbf4eaf6223bef4e8ce34d6bb12556c5f9249e", size = 11581098, upload-time = "2026-07-16T15:13:42.132Z" }, - { url = "https://files.pythonhosted.org/packages/f6/f9/a0d4871d12fae702eb1f41b686caf05f1f8b124dc6db6f784f53d74918fa/ruff-0.15.22-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:365523eb91d9224e1bcb03b022fbf0facb8f9e23792a2c53d9d4b3924bdbdebb", size = 11399422, upload-time = "2026-07-16T15:13:45.2Z" }, - { url = "https://files.pythonhosted.org/packages/18/80/c843a5176cddbceb0b7e8dd41cf9993490796c1c469348d384f5a5c13c56/ruff-0.15.22-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:fabfd168afdf29fee5be98b831efa9683c94d7c5a3b58b9ce5a2e38444589a74", size = 11381683, upload-time = "2026-07-16T15:13:48.46Z" }, - { url = "https://files.pythonhosted.org/packages/d4/00/8485de0ae92239438a36cfc51350db9b9e85c9ebdfaea91b18e422706662/ruff-0.15.22-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:225dbf095a87f1d9f90f5fd7924d2613ee452a75a4308c63a8f50f761787aa7c", size = 10850295, upload-time = "2026-07-16T15:13:51.655Z" }, - { url = "https://files.pythonhosted.org/packages/fa/91/24977ec2ec72eaf15e4394ace2959fdff2dd1e14f03e005e838023407169/ruff-0.15.22-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:1877d63b9d24ed278744f1523fd11b85540566d54641f97c566d7d9dc5ca5296", size = 10579640, upload-time = "2026-07-16T15:13:54.79Z" }, - { url = "https://files.pythonhosted.org/packages/9c/47/9b51216951974df1f263ac19da550d34252e0ed7218c25f10c5ef9ed7517/ruff-0.15.22-py3-none-musllinux_1_2_i686.whl", hash = "sha256:a1606c510bd7215680d32efab38965f7cdec3ef69f5170a3f4791404ffdd5262", size = 11105077, upload-time = "2026-07-16T15:13:57.915Z" }, - { url = "https://files.pythonhosted.org/packages/c2/47/20e9d4a3b8016778acea5fc32bb50d35d207500a17ddb529ffa6996feef8/ruff-0.15.22-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:630479b18625f5ffc373f77603a22a9f8ac0acd7ff0501178b5db28ec71e9c64", size = 11490980, upload-time = "2026-07-16T15:14:01.032Z" }, - { url = "https://files.pythonhosted.org/packages/4d/76/3f72d8fc38c1cb77b38c56a70da9d0c17700cc1cc50f9649c9d3c8f5ba71/ruff-0.15.22-py3-none-win32.whl", hash = "sha256:e5ba0e4a13fd14abbed2a77b517a3911290c6c6c59ef67784328d1668fab76cf", size = 10789165, upload-time = "2026-07-16T15:14:04.16Z" }, - { url = "https://files.pythonhosted.org/packages/cb/46/4965251734c2b6fcdca1b1b187d20bcac3af0ee5b083b89c910bb961ce3a/ruff-0.15.22-py3-none-win_amd64.whl", hash = "sha256:9be63ba1eb936acd2d1342fb8337c356353706fce233b2a15a09a97037e6acde", size = 11938297, upload-time = "2026-07-16T15:14:07.316Z" }, - { url = "https://files.pythonhosted.org/packages/57/c9/e69b1ff4c8b69093ef08b8919ab767af0569666865b39c30a8795d88d3c6/ruff-0.15.22-py3-none-win_arm64.whl", hash = "sha256:e1168075b72158510839f250027659cdd78476f40507dd517892304c41318661", size = 11298172, upload-time = "2026-07-16T15:14:10.51Z" }, +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4d/94/1e5e4967626faf12fa56999cd6222dff6992ceb086ad7945756baf70c7a7/ruff-0.16.0.tar.gz", hash = "sha256:e460aafd5495ec89efaa6ced2e4a9a581116451e1c88b9d37ef497e0f8e93982", size = 4790557, upload-time = "2026-07-23T19:11:30.981Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4b/81/1c8818fee7ce1a04cd7d1b3172e0a8f8e4f1dc4feb7fc390e16daa8af323/ruff-0.16.0-py3-none-linux_armv6l.whl", hash = "sha256:e5115729eb08c585e5121978ba5d5b60caeae394ce21b9fb5e6cd33a1c6c9b1e", size = 10754633, upload-time = "2026-07-23T19:10:46.415Z" }, + { url = "https://files.pythonhosted.org/packages/23/df/beaf59c09d68db84304d555f188b276a77132a5d5b0b67a5c762aa143628/ruff-0.16.0-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:3c954b1d580bfa035b41654f7858cc7e71d5fc3ac5b723dd62bd9133830ed522", size = 10969164, upload-time = "2026-07-23T19:10:50.271Z" }, + { url = "https://files.pythonhosted.org/packages/42/ce/741cd197496a1abbf51352710fd15ed995d2a2be87189c1da26a450d6e83/ruff-0.16.0-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e01c21d10eb1b29f47b7454e1f4056db9a3f0260c646aa88457c610291db9f81", size = 10488846, upload-time = "2026-07-23T19:10:52.639Z" }, + { url = "https://files.pythonhosted.org/packages/52/2a/a2db8e88cade358f5cdcb05674a917751074109315d014eb6352d9a893f7/ruff-0.16.0-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6e364e5ed22ed8dc05082fd78e35308618260907ac2d3c1d637b2e682415b6c9", size = 10889729, upload-time = "2026-07-23T19:10:54.89Z" }, + { url = "https://files.pythonhosted.org/packages/42/65/62a771694ebd63029dc953e27dbad40e1588bd4860ff9fe881018fddaa49/ruff-0.16.0-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d327b8fc113a1d4421a04f3839d3752057c8dd1ee320223a6f3f52d04ada462a", size = 10568275, upload-time = "2026-07-23T19:10:56.993Z" }, + { url = "https://files.pythonhosted.org/packages/3f/e2/ced249fe8af5f086c5c58cc21cc3356d50f32f7401c5df87050c999620a7/ruff-0.16.0-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:a9b50c55e263103586b3dcf5f73d479eb8cb5fdb6098fec59a62891dab653717", size = 11385112, upload-time = "2026-07-23T19:10:59.615Z" }, + { url = "https://files.pythonhosted.org/packages/87/0b/05154977a8fd69eeb6c103271f55403bfd8711f5c0f8ed07489d95a504e7/ruff-0.16.0-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0ff4a79ce3ec0172f3241943835de1c4cb4e2dcd07f0f8c2d02603dbbbee4b17", size = 12207008, upload-time = "2026-07-23T19:11:02.154Z" }, + { url = "https://files.pythonhosted.org/packages/fb/29/98225831a3a1eab0e02f4acc6ca6559a98611dcc68b6965ff4b7234627c1/ruff-0.16.0-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e95c448fca1fb2a18372a9440926c5a6ee789639bb975c72e7ae6d0b04218ab4", size = 11650842, upload-time = "2026-07-23T19:11:04.557Z" }, + { url = "https://files.pythonhosted.org/packages/91/66/6bd3cf90500653d55dc0ffc8507aa8300bd49d0214b2e8cb4d3fef2943ba/ruff-0.16.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4f11a8d11010301d0a398a2fdef67691feca7294da6aef55e2150e8fa2cd520b", size = 11400718, upload-time = "2026-07-23T19:11:09.233Z" }, + { url = "https://files.pythonhosted.org/packages/8e/a2/a54eb4eae05d66364050a5d3b8a9c5ef88196531b3cbe7109d873f87f819/ruff-0.16.0-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:48044c678e9cb8698246c99b14aaccfa6601dea7379eb48a6f8f73f7a6d86cd0", size = 11426177, upload-time = "2026-07-23T19:11:11.994Z" }, + { url = "https://files.pythonhosted.org/packages/1a/be/16e3eea4b2a478a496919f5e36f17c4559e54620bd3bbac5d6affa068006/ruff-0.16.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:7aa0959bad8eb8bef50340154fc9b58678dae31fa4293afa38b44b6e552c0213", size = 10856126, upload-time = "2026-07-23T19:11:14.221Z" }, + { url = "https://files.pythonhosted.org/packages/a2/84/252eb8b868a16eec7257c14f504f77537e734b2d69c762e639e588e304a3/ruff-0.16.0-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:28ea2b7df8ebf7f9da6b7d47b230ab48f387c0a29be3b474c4d0740e197bb9af", size = 10571208, upload-time = "2026-07-23T19:11:16.378Z" }, + { url = "https://files.pythonhosted.org/packages/21/09/817a482f542f7570cbb4554b26e896610c7114f539b1d9e2d2145bf6bef6/ruff-0.16.0-py3-none-musllinux_1_2_i686.whl", hash = "sha256:33a3dfac8c35f81498dea9181bccc2f4c4bc8f1521a1dd9406e77643e0f0fb09", size = 11063329, upload-time = "2026-07-23T19:11:19.173Z" }, + { url = "https://files.pythonhosted.org/packages/2e/23/9403c180ca1cb9b1f7335f5c3e5305c09d49ea5b345196682a36028bde4a/ruff-0.16.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:a5237a0bda500d30d81b8e07a6973a5cbc772864cbf746ae2f4e8a2e01c9f4ed", size = 11489751, upload-time = "2026-07-23T19:11:21.74Z" }, + { url = "https://files.pythonhosted.org/packages/b2/1d/1b2ef7bcde851c78d7f17f1cca13fd6dc695fc4b3d6197941e72cae5b132/ruff-0.16.0-py3-none-win32.whl", hash = "sha256:7fab76fa065c873f41ff744347c6e77bcc3dfec4bcc754dc26b63d23c0f7f5fb", size = 10785885, upload-time = "2026-07-23T19:11:23.947Z" }, + { url = "https://files.pythonhosted.org/packages/b2/a3/d5e4ef7a56be3f928ffb90b94c25ba7d3cb9c7fe0736aeaaedf361770712/ruff-0.16.0-py3-none-win_amd64.whl", hash = "sha256:429c117f022bf481fabd9d551e7a3952b24c65e6ef44337ea09d90bebef14472", size = 11923141, upload-time = "2026-07-23T19:11:26.409Z" }, + { url = "https://files.pythonhosted.org/packages/cb/9a/8415f2657cbe200f41a4531ccededf135505a92d4a012229121f885b26f9/ruff-0.16.0-py3-none-win_arm64.whl", hash = "sha256:14296fedcd2705c77ab8235439278bbb38f285cf7da5528b00b3e330c3d4872d", size = 11273407, upload-time = "2026-07-23T19:11:28.705Z" }, ] [[package]] @@ -21593,7 +21645,7 @@ wheels = [ [[package]] name = "scikit-learn" -version = "1.5.2" +version = "1.7.2" source = { registry = "https://pypi.org/simple" } resolution-markers = [ "python_full_version < '3.11' and platform_machine == 'arm64' and sys_platform == 'darwin'", @@ -21605,28 +21657,38 @@ dependencies = [ { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" } }, { name = "threadpoolctl" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/37/59/44985a2bdc95c74e34fef3d10cb5d93ce13b0e2a7baefffe1b53853b502d/scikit_learn-1.5.2.tar.gz", hash = "sha256:b4237ed7b3fdd0a4882792e68ef2545d5baa50aca3bb45aa7df468138ad8f94d", size = 7001680, upload-time = "2024-09-11T15:50:10.957Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/98/89/be41419b4bec629a4691183a5eb1796f91252a13a5ffa243fd958cad7e91/scikit_learn-1.5.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:299406827fb9a4f862626d0fe6c122f5f87f8910b86fe5daa4c32dcd742139b6", size = 12106070, upload-time = "2024-09-11T15:49:19.633Z" }, - { url = "https://files.pythonhosted.org/packages/bf/e0/3b6d777d375f3b685f433c93384cdb724fb078e1dc8f8ff0950467e56c30/scikit_learn-1.5.2-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:2d4cad1119c77930b235579ad0dc25e65c917e756fe80cab96aa3b9428bd3fb0", size = 10971758, upload-time = "2024-09-11T15:49:22.484Z" }, - { url = "https://files.pythonhosted.org/packages/7b/31/eb7dd56c371640753953277de11356c46a3149bfeebb3d7dcd90b993715a/scikit_learn-1.5.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8c412ccc2ad9bf3755915e3908e677b367ebc8d010acbb3f182814524f2e5540", size = 12500080, upload-time = "2024-09-11T15:49:24.975Z" }, - { url = "https://files.pythonhosted.org/packages/4c/1e/a7c7357e704459c7d56a18df4a0bf08669442d1f8878cc0864beccd6306a/scikit_learn-1.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a686885a4b3818d9e62904d91b57fa757fc2bed3e465c8b177be652f4dd37c8", size = 13347241, upload-time = "2024-09-11T15:49:27.891Z" }, - { url = "https://files.pythonhosted.org/packages/48/76/154ebda6794faf0b0f3ccb1b5cd9a19f0a63cb9e1f3d2c61b6114002677b/scikit_learn-1.5.2-cp310-cp310-win_amd64.whl", hash = "sha256:c15b1ca23d7c5f33cc2cb0a0d6aaacf893792271cddff0edbd6a40e8319bc113", size = 11000477, upload-time = "2024-09-11T15:49:30.693Z" }, - { url = "https://files.pythonhosted.org/packages/ff/91/609961972f694cb9520c4c3d201e377a26583e1eb83bc5a334c893729214/scikit_learn-1.5.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:03b6158efa3faaf1feea3faa884c840ebd61b6484167c711548fce208ea09445", size = 12088580, upload-time = "2024-09-11T15:49:33.55Z" }, - { url = "https://files.pythonhosted.org/packages/cd/7a/19fe32c810c5ceddafcfda16276d98df299c8649e24e84d4f00df4a91e01/scikit_learn-1.5.2-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:1ff45e26928d3b4eb767a8f14a9a6efbf1cbff7c05d1fb0f95f211a89fd4f5de", size = 10975994, upload-time = "2024-09-11T15:49:35.728Z" }, - { url = "https://files.pythonhosted.org/packages/4c/75/62e49f8a62bf3c60b0e64d0fce540578ee4f0e752765beb2e1dc7c6d6098/scikit_learn-1.5.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f763897fe92d0e903aa4847b0aec0e68cadfff77e8a0687cabd946c89d17e675", size = 12465782, upload-time = "2024-09-11T15:49:38.596Z" }, - { url = "https://files.pythonhosted.org/packages/49/21/3723de321531c9745e40f1badafd821e029d346155b6c79704e0b7197552/scikit_learn-1.5.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f8b0ccd4a902836493e026c03256e8b206656f91fbcc4fde28c57a5b752561f1", size = 13322034, upload-time = "2024-09-11T15:49:41.452Z" }, - { url = "https://files.pythonhosted.org/packages/17/1c/ccdd103cfcc9435a18819856fbbe0c20b8fa60bfc3343580de4be13f0668/scikit_learn-1.5.2-cp311-cp311-win_amd64.whl", hash = "sha256:6c16d84a0d45e4894832b3c4d0bf73050939e21b99b01b6fd59cbb0cf39163b6", size = 11015224, upload-time = "2024-09-11T15:49:43.692Z" }, - { url = "https://files.pythonhosted.org/packages/a4/db/b485c1ac54ff3bd9e7e6b39d3cc6609c4c76a65f52ab0a7b22b6c3ab0e9d/scikit_learn-1.5.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:f932a02c3f4956dfb981391ab24bda1dbd90fe3d628e4b42caef3e041c67707a", size = 12110344, upload-time = "2024-09-11T15:49:46.253Z" }, - { url = "https://files.pythonhosted.org/packages/54/1a/7deb52fa23aebb855431ad659b3c6a2e1709ece582cb3a63d66905e735fe/scikit_learn-1.5.2-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:3b923d119d65b7bd555c73be5423bf06c0105678ce7e1f558cb4b40b0a5502b1", size = 11033502, upload-time = "2024-09-11T15:49:48.656Z" }, - { url = "https://files.pythonhosted.org/packages/a1/32/4a7a205b14c11225609b75b28402c196e4396ac754dab6a81971b811781c/scikit_learn-1.5.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f60021ec1574e56632be2a36b946f8143bf4e5e6af4a06d85281adc22938e0dd", size = 12085794, upload-time = "2024-09-11T15:49:51.388Z" }, - { url = "https://files.pythonhosted.org/packages/c6/29/044048c5e911373827c0e1d3051321b9183b2a4f8d4e2f11c08fcff83f13/scikit_learn-1.5.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:394397841449853c2290a32050382edaec3da89e35b3e03d6cc966aebc6a8ae6", size = 12945797, upload-time = "2024-09-11T15:49:53.579Z" }, - { url = "https://files.pythonhosted.org/packages/aa/ce/c0b912f2f31aeb1b756a6ba56bcd84dd1f8a148470526a48515a3f4d48cd/scikit_learn-1.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:57cc1786cfd6bd118220a92ede80270132aa353647684efa385a74244a41e3b1", size = 10985467, upload-time = "2024-09-11T15:49:56.446Z" }, - { url = "https://files.pythonhosted.org/packages/a4/50/8891028437858cc510e13578fe7046574a60c2aaaa92b02d64aac5b1b412/scikit_learn-1.5.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e9a702e2de732bbb20d3bad29ebd77fc05a6b427dc49964300340e4c9328b3f5", size = 12025584, upload-time = "2024-10-02T18:35:29.369Z" }, - { url = "https://files.pythonhosted.org/packages/d2/79/17feef8a1c14149436083bec0e61d7befb4812e272d5b20f9d79ea3e9ab1/scikit_learn-1.5.2-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:b0768ad641981f5d3a198430a1d31c3e044ed2e8a6f22166b4d546a5116d7908", size = 10959795, upload-time = "2024-10-02T18:35:34.22Z" }, - { url = "https://files.pythonhosted.org/packages/b1/c8/f08313f9e2e656bd0905930ae8bf99a573ea21c34666a813b749c338202f/scikit_learn-1.5.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:178ddd0a5cb0044464fc1bfc4cca5b1833bfc7bb022d70b05db8530da4bb3dd3", size = 12077302, upload-time = "2024-10-02T18:35:38.911Z" }, - { url = "https://files.pythonhosted.org/packages/a7/48/fbfb4dc72bed0fe31fe045fb30e924909ad03f717c36694351612973b1a9/scikit_learn-1.5.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f7284ade780084d94505632241bf78c44ab3b6f1e8ccab3d2af58e0e950f9c12", size = 13002811, upload-time = "2024-10-02T18:35:43.28Z" }, - { url = "https://files.pythonhosted.org/packages/a5/e7/0c869f9e60d225a77af90d2aefa7a4a4c0e745b149325d1450f0f0ce5399/scikit_learn-1.5.2-cp313-cp313-win_amd64.whl", hash = "sha256:b7b0f9a0b1040830d38c39b91b3a44e1b643f4b36e36567b80b7c6bd2202a27f", size = 10951354, upload-time = "2024-10-02T18:35:47.954Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/98/c2/a7855e41c9d285dfe86dc50b250978105dce513d6e459ea66a6aeb0e1e0c/scikit_learn-1.7.2.tar.gz", hash = "sha256:20e9e49ecd130598f1ca38a1d85090e1a600147b9c02fa6f15d69cb53d968fda", size = 7193136, upload-time = "2025-09-09T08:21:29.075Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/3e/daed796fd69cce768b8788401cc464ea90b306fb196ae1ffed0b98182859/scikit_learn-1.7.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6b33579c10a3081d076ab403df4a4190da4f4432d443521674637677dc91e61f", size = 9336221, upload-time = "2025-09-09T08:20:19.328Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ce/af9d99533b24c55ff4e18d9b7b4d9919bbc6cd8f22fe7a7be01519a347d5/scikit_learn-1.7.2-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:36749fb62b3d961b1ce4fedf08fa57a1986cd409eff2d783bca5d4b9b5fce51c", size = 8653834, upload-time = "2025-09-09T08:20:22.073Z" }, + { url = "https://files.pythonhosted.org/packages/58/0e/8c2a03d518fb6bd0b6b0d4b114c63d5f1db01ff0f9925d8eb10960d01c01/scikit_learn-1.7.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7a58814265dfc52b3295b1900cfb5701589d30a8bb026c7540f1e9d3499d5ec8", size = 9660938, upload-time = "2025-09-09T08:20:24.327Z" }, + { url = "https://files.pythonhosted.org/packages/2b/75/4311605069b5d220e7cf5adabb38535bd96f0079313cdbb04b291479b22a/scikit_learn-1.7.2-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a847fea807e278f821a0406ca01e387f97653e284ecbd9750e3ee7c90347f18", size = 9477818, upload-time = "2025-09-09T08:20:26.845Z" }, + { url = "https://files.pythonhosted.org/packages/7f/9b/87961813c34adbca21a6b3f6b2bea344c43b30217a6d24cc437c6147f3e8/scikit_learn-1.7.2-cp310-cp310-win_amd64.whl", hash = "sha256:ca250e6836d10e6f402436d6463d6c0e4d8e0234cfb6a9a47835bd392b852ce5", size = 8886969, upload-time = "2025-09-09T08:20:29.329Z" }, + { url = "https://files.pythonhosted.org/packages/43/83/564e141eef908a5863a54da8ca342a137f45a0bfb71d1d79704c9894c9d1/scikit_learn-1.7.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c7509693451651cd7361d30ce4e86a1347493554f172b1c72a39300fa2aea79e", size = 9331967, upload-time = "2025-09-09T08:20:32.421Z" }, + { url = "https://files.pythonhosted.org/packages/18/d6/ba863a4171ac9d7314c4d3fc251f015704a2caeee41ced89f321c049ed83/scikit_learn-1.7.2-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:0486c8f827c2e7b64837c731c8feff72c0bd2b998067a8a9cbc10643c31f0fe1", size = 8648645, upload-time = "2025-09-09T08:20:34.436Z" }, + { url = "https://files.pythonhosted.org/packages/ef/0e/97dbca66347b8cf0ea8b529e6bb9367e337ba2e8be0ef5c1a545232abfde/scikit_learn-1.7.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:89877e19a80c7b11a2891a27c21c4894fb18e2c2e077815bcade10d34287b20d", size = 9715424, upload-time = "2025-09-09T08:20:36.776Z" }, + { url = "https://files.pythonhosted.org/packages/f7/32/1f3b22e3207e1d2c883a7e09abb956362e7d1bd2f14458c7de258a26ac15/scikit_learn-1.7.2-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8da8bf89d4d79aaec192d2bda62f9b56ae4e5b4ef93b6a56b5de4977e375c1f1", size = 9509234, upload-time = "2025-09-09T08:20:38.957Z" }, + { url = "https://files.pythonhosted.org/packages/9f/71/34ddbd21f1da67c7a768146968b4d0220ee6831e4bcbad3e03dd3eae88b6/scikit_learn-1.7.2-cp311-cp311-win_amd64.whl", hash = "sha256:9b7ed8d58725030568523e937c43e56bc01cadb478fc43c042a9aca1dacb3ba1", size = 8894244, upload-time = "2025-09-09T08:20:41.166Z" }, + { url = "https://files.pythonhosted.org/packages/a7/aa/3996e2196075689afb9fce0410ebdb4a09099d7964d061d7213700204409/scikit_learn-1.7.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8d91a97fa2b706943822398ab943cde71858a50245e31bc71dba62aab1d60a96", size = 9259818, upload-time = "2025-09-09T08:20:43.19Z" }, + { url = "https://files.pythonhosted.org/packages/43/5d/779320063e88af9c4a7c2cf463ff11c21ac9c8bd730c4a294b0000b666c9/scikit_learn-1.7.2-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:acbc0f5fd2edd3432a22c69bed78e837c70cf896cd7993d71d51ba6708507476", size = 8636997, upload-time = "2025-09-09T08:20:45.468Z" }, + { url = "https://files.pythonhosted.org/packages/5c/d0/0c577d9325b05594fdd33aa970bf53fb673f051a45496842caee13cfd7fe/scikit_learn-1.7.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e5bf3d930aee75a65478df91ac1225ff89cd28e9ac7bd1196853a9229b6adb0b", size = 9478381, upload-time = "2025-09-09T08:20:47.982Z" }, + { url = "https://files.pythonhosted.org/packages/82/70/8bf44b933837ba8494ca0fc9a9ab60f1c13b062ad0197f60a56e2fc4c43e/scikit_learn-1.7.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b4d6e9deed1a47aca9fe2f267ab8e8fe82ee20b4526b2c0cd9e135cea10feb44", size = 9300296, upload-time = "2025-09-09T08:20:50.366Z" }, + { url = "https://files.pythonhosted.org/packages/c6/99/ed35197a158f1fdc2fe7c3680e9c70d0128f662e1fee4ed495f4b5e13db0/scikit_learn-1.7.2-cp312-cp312-win_amd64.whl", hash = "sha256:6088aa475f0785e01bcf8529f55280a3d7d298679f50c0bb70a2364a82d0b290", size = 8731256, upload-time = "2025-09-09T08:20:52.627Z" }, + { url = "https://files.pythonhosted.org/packages/ae/93/a3038cb0293037fd335f77f31fe053b89c72f17b1c8908c576c29d953e84/scikit_learn-1.7.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0b7dacaa05e5d76759fb071558a8b5130f4845166d88654a0f9bdf3eb57851b7", size = 9212382, upload-time = "2025-09-09T08:20:54.731Z" }, + { url = "https://files.pythonhosted.org/packages/40/dd/9a88879b0c1104259136146e4742026b52df8540c39fec21a6383f8292c7/scikit_learn-1.7.2-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:abebbd61ad9e1deed54cca45caea8ad5f79e1b93173dece40bb8e0c658dbe6fe", size = 8592042, upload-time = "2025-09-09T08:20:57.313Z" }, + { url = "https://files.pythonhosted.org/packages/46/af/c5e286471b7d10871b811b72ae794ac5fe2989c0a2df07f0ec723030f5f5/scikit_learn-1.7.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:502c18e39849c0ea1a5d681af1dbcf15f6cce601aebb657aabbfe84133c1907f", size = 9434180, upload-time = "2025-09-09T08:20:59.671Z" }, + { url = "https://files.pythonhosted.org/packages/f1/fd/df59faa53312d585023b2da27e866524ffb8faf87a68516c23896c718320/scikit_learn-1.7.2-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a4c328a71785382fe3fe676a9ecf2c86189249beff90bf85e22bdb7efaf9ae0", size = 9283660, upload-time = "2025-09-09T08:21:01.71Z" }, + { url = "https://files.pythonhosted.org/packages/a7/c7/03000262759d7b6f38c836ff9d512f438a70d8a8ddae68ee80de72dcfb63/scikit_learn-1.7.2-cp313-cp313-win_amd64.whl", hash = "sha256:63a9afd6f7b229aad94618c01c252ce9e6fa97918c5ca19c9a17a087d819440c", size = 8702057, upload-time = "2025-09-09T08:21:04.234Z" }, + { url = "https://files.pythonhosted.org/packages/55/87/ef5eb1f267084532c8e4aef98a28b6ffe7425acbfd64b5e2f2e066bc29b3/scikit_learn-1.7.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:9acb6c5e867447b4e1390930e3944a005e2cb115922e693c08a323421a6966e8", size = 9558731, upload-time = "2025-09-09T08:21:06.381Z" }, + { url = "https://files.pythonhosted.org/packages/93/f8/6c1e3fc14b10118068d7938878a9f3f4e6d7b74a8ddb1e5bed65159ccda8/scikit_learn-1.7.2-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:2a41e2a0ef45063e654152ec9d8bcfc39f7afce35b08902bfe290c2498a67a6a", size = 9038852, upload-time = "2025-09-09T08:21:08.628Z" }, + { url = "https://files.pythonhosted.org/packages/83/87/066cafc896ee540c34becf95d30375fe5cbe93c3b75a0ee9aa852cd60021/scikit_learn-1.7.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:98335fb98509b73385b3ab2bd0639b1f610541d3988ee675c670371d6a87aa7c", size = 9527094, upload-time = "2025-09-09T08:21:11.486Z" }, + { url = "https://files.pythonhosted.org/packages/9c/2b/4903e1ccafa1f6453b1ab78413938c8800633988c838aa0be386cbb33072/scikit_learn-1.7.2-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:191e5550980d45449126e23ed1d5e9e24b2c68329ee1f691a3987476e115e09c", size = 9367436, upload-time = "2025-09-09T08:21:13.602Z" }, + { url = "https://files.pythonhosted.org/packages/b5/aa/8444be3cfb10451617ff9d177b3c190288f4563e6c50ff02728be67ad094/scikit_learn-1.7.2-cp313-cp313t-win_amd64.whl", hash = "sha256:57dc4deb1d3762c75d685507fbd0bc17160144b2f2ba4ccea5dc285ab0d0e973", size = 9275749, upload-time = "2025-09-09T08:21:15.96Z" }, + { url = "https://files.pythonhosted.org/packages/d9/82/dee5acf66837852e8e68df6d8d3a6cb22d3df997b733b032f513d95205b7/scikit_learn-1.7.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fa8f63940e29c82d1e67a45d5297bdebbcb585f5a5a50c4914cc2e852ab77f33", size = 9208906, upload-time = "2025-09-09T08:21:18.557Z" }, + { url = "https://files.pythonhosted.org/packages/3c/30/9029e54e17b87cb7d50d51a5926429c683d5b4c1732f0507a6c3bed9bf65/scikit_learn-1.7.2-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:f95dc55b7902b91331fa4e5845dd5bde0580c9cd9612b1b2791b7e80c3d32615", size = 8627836, upload-time = "2025-09-09T08:21:20.695Z" }, + { url = "https://files.pythonhosted.org/packages/60/18/4a52c635c71b536879f4b971c2cedf32c35ee78f48367885ed8025d1f7ee/scikit_learn-1.7.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9656e4a53e54578ad10a434dc1f993330568cfee176dff07112b8785fb413106", size = 9426236, upload-time = "2025-09-09T08:21:22.645Z" }, + { url = "https://files.pythonhosted.org/packages/99/7e/290362f6ab582128c53445458a5befd471ed1ea37953d5bcf80604619250/scikit_learn-1.7.2-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96dc05a854add0e50d3f47a1ef21a10a595016da5b007c7d9cd9d0bffd1fcc61", size = 9312593, upload-time = "2025-09-09T08:21:24.65Z" }, + { url = "https://files.pythonhosted.org/packages/8e/87/24f541b6d62b1794939ae6422f8023703bbf6900378b2b34e0b4384dfefd/scikit_learn-1.7.2-cp314-cp314-win_amd64.whl", hash = "sha256:bb24510ed3f9f61476181e4db51ce801e2ba37541def12dc9333b946fc7a9cf8", size = 8820007, upload-time = "2025-09-09T08:21:26.713Z" }, ] [[package]] @@ -22143,6 +22205,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/76/d3/250e13e7b3473a0f441f4d680d511ad30ce432d0180115dec9840d88fe89/smbprotocol-1.17.0-py3-none-any.whl", hash = "sha256:bd1abff5417f5af83ca516a64ab8e5acece3dbdcf58d4e5e23e47f5165a77349", size = 128208, upload-time = "2026-07-07T03:23:19.178Z" }, ] +[package.optional-dependencies] +kerberos = [ + { name = "pyspnego", extra = ["kerberos"] }, +] + [[package]] name = "smmap" version = "5.0.3"