Skip to content

Vendor the Java SDK supervisor schema from this repository - #75942

Open
jason810496 wants to merge 12 commits into
apache:mainfrom
jason810496:feature/java-sdk-local-supervisor-schema
Open

jason810496 wants to merge 12 commits into
apache:mainfrom
jason810496:feature/java-sdk-local-supervisor-schema

Conversation

@jason810496

@jason810496 jason810496 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

The Java SDK generates its supervisor message classes from a vendored copy of the supervisor schema, refreshed by downloading the version in java-sdk/gradle.properties from the staged docs site. That had two problems:

  • New versions cannot be fetched. The site only publishes released versions, so the download 404s for an in-progress one.
  • Changes without an api_version bump were missed. The hook compared versions, not content.

The vendored copy now comes from the Task SDK schema in this repository. airflowSupervisorSchemaVersion must be a version the Task SDK schema bundle defines:

  • When it is the newest one, the copy is compared by content and refreshed from the Task SDK schema, like sync-go-sdk-schemas. The hook also runs when the Task SDK schema changes.
  • When it is older, the copy must already declare that version, as copied from the release that ships it, and is left alone.
$ ./scripts/ci/prek/sync_java_sdk_supervisor_schema.py
Refreshed java-sdk/sdk/schema/schema.json from task-sdk/src/airflow/sdk/execution_time/schema/schema.json.
Review the diff, then git add it and recommit.

:sdk:syncSupervisorSchema follows the same rules and no longer downloads anything. Outside the monorepo it fails if the vendored copy does not declare the pinned version, so a source release builds from the same schema as an in-repo build, without network access.

The Java SDK stays on supervisor schema 2026-10-30. The vendored copy was behind what Airflow 3.4 ships for that version, so it is refreshed from the Task SDK schema in 3.4.0rc3.

@jason810496
jason810496 force-pushed the feature/java-sdk-local-supervisor-schema branch 2 times, most recently from aff20b7 to 7af6b46 Compare November 2, 2026 09:09
@jason810496
jason810496 marked this pull request as ready for review November 2, 2026 09:53
@jason810496
jason810496 force-pushed the feature/java-sdk-local-supervisor-schema branch from 7af6b46 to f4024e3 Compare November 3, 2026 02:11
jason810496 and others added 2 commits November 3, 2026 02:18
The vendored copy was refreshed by downloading the version named in
gradle.properties. The remote only publishes released versions, so an
in-progress one cannot be fetched, and a schema change that leaves
api_version alone moved nothing, so the copy went stale unnoticed.

It is now copied from the schema the Python Task SDK owns and compared by
content, the way sync-go-sdk-schemas already does for the Go SDK, so any
change to the source moves it in the same commit. The download stays as the
fallback for a build with no monorepo around it, which is what the source
distribution needs.

The refreshed copy is the content it never received while it was refreshed
by version; api_version is unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
The pin is the supervisor schema version the Java SDK speaks; the vendored
schema.json is a copy of whatever the bundle carries. Requiring the two to be
equal made the pin follow main every time a supervisor version was cut.

The prek hook now copies the source by content and only checks that the pin is
a version the bundle defines, and the Gradle task copies the in-repo source
whenever it exists instead of falling back to the download on a version
mismatch, so the build and the hook agree.

Co-Authored-By: Claude <noreply@anthropic.com>
@jason810496
jason810496 force-pushed the feature/java-sdk-local-supervisor-schema branch from f4024e3 to 917cca8 Compare November 3, 2026 02:21
@uranusjr uranusjr removed the backport-to-v3-4-test Backport to v3-4-test label Nov 5, 2026
@uranusjr uranusjr added the backport-to-v3-4-test Backport to v3-4-test label Nov 5, 2026

@uranusjr uranusjr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worked on the PR abit since it’s needed for another fix I want to get in. (Not strictly, just making things easier.) Please take a look @jason810496

@jason810496

Copy link
Copy Markdown
Member Author

I worked on the PR abit since it’s needed for another fix I want to get in. (Not strictly, just making things easier.) Please take a look @jason810496

Thanks, I will merge this after the Java SDK 1.0 is out to keep the Java SDK schema clean on v3-4-test.
Just in case of unexpected side effect of this PR.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants