Skip to content

Make Vertex AI evaluation stack an opt-in provider extra - #69385

Closed
YAshhh29 wants to merge 1 commit into
apache:mainfrom
YAshhh29:fix-google-aiplatform-evaluation-extra
Closed

YAshhh29 wants to merge 1 commit into
apache:mainfrom
YAshhh29:fix-google-aiplatform-evaluation-extra

Conversation

@YAshhh29

@YAshhh29 YAshhh29 commented Jul 4, 2026 •

Copy link
Copy Markdown
Contributor

Why

The Google provider enabled google-cloud-aiplatform[evaluation] for every installation, even though its evaluation dependency stack is only needed by Vertex AI generative-model evaluation APIs. Users of unrelated Google hooks and operators therefore received unnecessary ML and evaluation dependencies.

What changed

  • Keep google-cloud-aiplatform>=1.164.0 in the base provider dependencies without enabling evaluation.
  • Add a vertex-eval provider extra that installs google-cloud-aiplatform[evaluation]>=1.164.0.
  • Load vertexai.preview.evaluation only when an evaluation API is called and provide an actionable installation error when it is unavailable.
  • Update generated dependency tables and add a breaking-change migration note.
  • Add regressions for the dependency metadata split and lazy evaluation import.

Migration

Users of GenerativeModelHook.get_eval_task, GenerativeModelHook.run_evaluation, or RunEvaluationOperator must install:

pip install 'apache-airflow-providers-google[vertex-eval]'

Validation

  • Dependency metadata regression: 1 passed; the same assertions fail against upstream/main.
  • uv lock --check: passed, resolving 965 packages.
  • ruff format and ruff check: passed.
  • git diff --check: passed.
  • GitHub mergeability check: passed after rebasing onto current main.

Full provider tests were not run locally because Airflow does not support native Windows; CI provides the Linux test result.

related: #69323


Was generative AI tooling used to co-author this PR?
  • Yes - GitHub Copilot

Generated-by: GitHub Copilot following the guidelines


Drafted-by: GitHub Copilot (no human review before posting)

@YAshhh29
YAshhh29 requested a review from shahar1 as a code owner July 4, 2026 15:58
@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Jul 4, 2026
@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch 2 times, most recently from 300f1cc to 7ea92a2 Compare July 11, 2026 18:02
@YAshhh29

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main and fixed the Static Checks failure: the sync-provider-readme hook wanted to update the Requirements table in README.rst from google-cloud-aiplatform[evaluation] → google-cloud-aiplatform to match the new pyproject.toml base dep. I'd missed that during the initial edit — the docs index.rst had the right value but the README's Requirements table hadn't been re-synced.

The remaining core-DB test failures on the previous run (MySQL/Postgres/Sqlite/SQLAlchemy variants) don't touch anything in this diff and looked like a main-wide flake; the rebase should clear them.


Drafted-by: GitHub Copilot (Claude Opus 4.6); reviewed by @YAshhh29 before posting

@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 7ea92a2 to 9f5653e Compare July 12, 2026 10:04
@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 2dec77b to 1da574c Compare July 23, 2026 15:14
@YAshhh29

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main - the earlier core-DB failures were a stale-base flake and are now cleared, so all checks are green.
This is a small, self-contained dependency change (moves the heavy google-cloud-aiplatform[evaluation] stack behind an opt-in vertex-eval extra).
Would appreciate a review from a Google-provider maintainer when convenient. Thanks!

@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch 3 times, most recently from 7fec9ae to 67adb89 Compare July 27, 2026 18:55
@potiuk potiuk added the ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green. label Jul 28, 2026
@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 67adb89 to 69069e5 Compare August 12, 2026 19:47
@github-actions

github-actions Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

uv.lock on main just moved via #71924 ("[main] Upgrade important CI environment"), commit 4a247b9 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 69069e5 to 27b5e6d Compare August 14, 2026 10:31
@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 27b5e6d to 14662c7 Compare August 23, 2026 09:47
@YAshhh29

Copy link
Copy Markdown
Contributor Author

Quick update: I rebased this PR onto the latest main and resolved the changelog conflict.

While rebasing, I also tightened the regression coverage around the behavior this PR changes.
The tests now verify that Vertex AI evaluation dependencies remain opt-in through vertex-eval, and that the hook can still be imported without the evaluation stack installed.

uv lock --check, Ruff, and the focused metadata test all pass. GitHub now marks the PR as mergeable.
A fresh review would be appreciated.

@YAshhh29
YAshhh29 force-pushed the fix-google-aiplatform-evaluation-extra branch from 14662c7 to f89cc0c Compare August 23, 2026 14:40
@potiuk potiuk added the closed because of open PR limit Closed as a one-time step of introducing the open pull request limit label Sep 25, 2026
@potiuk

potiuk commented Sep 25, 2026

Copy link
Copy Markdown
Member

Hello @YAshhh29 - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 6 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers closed because of open PR limit Closed as a one-time step of introducing the open pull request limit kind:documentation provider:google Google (including GCP) related issues ready for maintainer review Set by triage when all criteria pass, or by the fork CI reconciler when fork CI is green.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants