Skip to content

Add OIDC worker token verification to Edge3 - #72262

Merged
ashb merged 2 commits into
apache:mainfrom
robertpofuk:edge3-oidc-worker-token-auth
Sep 24, 2026
Merged

ashb merged 2 commits into
apache:mainfrom
robertpofuk:edge3-oidc-worker-token-auth

Conversation

@robertpofuk

@robertpofuk robertpofuk commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Why

The Edge worker API currently only verifies worker tokens signed with the shared [api_auth] jwt_secret (symmetric HS512). This adds optional verification against a trusted OpenID Connect provider, so edge workers can authenticate with tokens minted by an external identity provider instead of the shared secret.

What

When [edge] trusted_jwks_url is set, worker tokens are verified against the provider JWKS using asymmetric signatures, checking the iss and (optionally) aud claims, and then an authorization hook decides whether the identity may act as an edge worker. New [edge] options:

  • trusted_jwks_url — provider JWKS URL; enables OIDC verification when set. Enabling it disables the per-request signed method claim check (externally issued tokens do not carry it), so authorization is delegated to jwt_verifier.
  • jwt_issuer — expected iss claim. Leave empty to skip issuer verification; set it to bind tokens to your provider.
  • jwt_audience — expected aud claim. When set, the token must carry a matching aud; when empty, only tokens without an aud claim are accepted (a token that carries an aud is rejected).
  • jwt_algorithm — accepted signing algorithms, comma-separated (default RS256).
  • jwt_leeway — clock-skew leeway in seconds for OIDC tokens (default 30).
  • jwt_verifier — import path of a callable verify(claims: dict, context: WorkerTokenContext) -> {"authorized": bool} run after signature/issuer/audience checks. It receives the validated claims and a request context carrying the request method. A falsy result or a raised exception rejects the request. Use it to restrict which identities (e.g. by sub, client_id, or scope) may act as edge workers. When empty, any validly signed token from the trusted issuer is authorized.

When trusted_jwks_url is empty (the default), the existing shared-secret verification is used unchanged, so existing deployments are unaffected.

Uses the existing core JWKS / JWTValidator. The core JWTValidator.audience annotation is widened to str | Sequence[str] | None (the runtime already accepts None) so an empty edge audience can be passed honestly instead of via a cast.

@boring-cyborg boring-cyborg Bot added area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3 labels Aug 29, 2026
@boring-cyborg

boring-cyborg Bot commented Aug 29, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch from f4c978c to 94c638d Compare August 29, 2026 20:56
@robertpofuk

Copy link
Copy Markdown
Contributor Author

Tested on our internal environment by vendoring edge3 and re-building. Works nicely and makes our setup super simple since workers just order therir access via central IDP.

@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 2 times, most recently from 0be8d57 to 203ac0c Compare August 30, 2026 18:33
@robertpofuk

Copy link
Copy Markdown
Contributor Author

Fixed CI failures and rebased on main.

Comment thread providers/edge3/provider.yaml Resolved
@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 2 times, most recently from bc32bf5 to 7365323 Compare September 7, 2026 04:08

@ashb ashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small nit

@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 3 times, most recently from 36c8d1c to a14db06 Compare September 8, 2026 19:02
@robertpofuk
robertpofuk requested a review from ashb September 9, 2026 09:52

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the shared-secret path is left genuinely untouched when trusted_jwks_url is empty, the config surface is well documented option-by-option, version_added: 4.4.0 is right off 4.3.1, and the tests are better than most: an autouse fixture clears the @cache between cases, and there is even an explicit test pinning the cache-stickiness behaviour. The [edge] section choice is correct too — the rest of edge3 reads from it.

This got two independent reviews: my own pass, and an adversarial second read. They converged on the same place, so I'm requesting changes rather than commenting.

To be clear about calibration first: nothing here breaks Airflow's documented security model. The feature is opt-in and a Deployment Manager has to enable it. What follows is about what the new code does once they do.

A trusted signature is not worker authorization

Details inline on auth.py:119. Today the signed method claim is the only thing that narrows what a worker token can do — a token minted for one endpoint cannot be replayed against another. OIDC mode drops that check and puts nothing in its place: no subject, no client, no scope restriction.

What is left is "this token was signed by a key in the configured JWKS, and it has iat and exp". That proves who signed the token; it does not establish that the identity is allowed to act as an edge worker. If the issuer serves any other workload — the normal case for a corporate IdP — an unrelated identity holding an aud-less token can register workers, fetch queued jobs, and change job states, since this function is the sole authentication dependency on those routes. That is strictly more access than a holder of the current shared secret has.

It compounds with the issuer default: [edge] jwt_issuer has no default, so the documented minimal configuration (trusted_jwks_url and nothing else) passes issuer=None, and PyJWT's _validate_iss returns immediately on None without checking anything. Enabling the feature in its simplest form therefore binds neither the issuer nor the identity.

What I'd like to see: an explicit worker-authorization policy driven by validated identity or scope claims, jwt_issuer required (failing closed at validator construction) whenever trusted_jwks_url is set, and an Edge-specific audience where the provider supports one.

The audience option documents behaviour the code does not have

Details inline on auth.py:81-82. jwt_audience empty means audience=None, and JWTValidator leaves PyJWT's verify_aud enabled. PyJWT's _validate_aud returns early only when the token has no aud; a token that carries one raises InvalidAudienceError.

So the config help — "leave empty to skip audience verification" — describes the opposite of what happens. An operator who follows it with an ordinary IdP token gets 403s and no obvious reason why. Either require a configured audience for tokens that carry aud and reword the option to say that empty accepts only audience-less tokens, or implement genuine skipping explicitly.

The tests cannot see either of the above

Both findings live in how a signed token is validated, and no test validates one — the suite asserts validator attributes (validator.jwks is not None, validator.algorithm == ["RS512"]) and calls _check_method_claim with hand-built dicts. That is real coverage of the method-claim branch, and I don't want to undersell it, but it cannot catch an audience rejection or an unauthorized-identity acceptance.

Signed-token tests would close this: one proving a token from the trusted issuer belonging to an unrelated identity is rejected, and a pair covering tokens with and without aud under both an empty and a configured jwt_audience.

Smaller observations

  • auth.py:119 — "is OIDC enabled?" is decided in two places with different lifetimes. jwt_validator() is @cached so the mode is fixed at first use, while _check_method_claim re-reads config on every request. They can disagree: a validator cached in shared-secret mode plus a config that now reports a JWKS URL gives shared-secret validation with the method check disabled — a shared-secret token valid for every method. Narrow, but this is the auth path, and one source of truth is cheap.
  • provider.yaml:123 — the config reference never mentions that enabling trusted_jwks_url removes per-method token scoping, and jwt_issuer's description does not say that leaving it unset means the issuer is not checked. Those are the two security-relevant consequences and they currently live only in the PR description.
  • The PR description documents oidc_jwks_url / oidc_issuer / oidc_audience / oidc_algorithms, but after the rename the keys are trusted_jwks_url / jwt_issuer / jwt_audience / jwt_algorithm. Provider changelogs are regenerated from git history, so the stale names can reach release notes.
  • Dropping nbf from required_claims relative to the core default is sensible for IdP-issued tokens — noting it only so the choice is on the record.

This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. After you've
addressed the points above and pushed an update, an Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Comment thread providers/edge3/provider.yaml Resolved
@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch from a14db06 to f9bbd44 Compare September 10, 2026 13:26
@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 2 times, most recently from 6cca294 to 0d37eb3 Compare September 10, 2026 13:43
@robertpofuk

robertpofuk commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Adjusted implementation to ensure that at least some assertions are required for checking weather token is allowed to be used with oidc enabled.

jwt_verifier concept is inspured by [core].hostname_callable so should fit overall style

jwt_verifier is on purpose built to return dict back. This way future requirements can be added. For exmaple core Airflow hasfunction to get_teams (i think in executor impelemtations). So We can use this return as place to extend and return from jwt_verifier {authorized: true teams [x, y]}. Much cleaner to have compatible extension if we return dict then fixed values.

@ashb
ashb requested a review from potiuk September 10, 2026 16:07
@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 2 times, most recently from 0625cd1 to f8ae22a Compare September 11, 2026 04:19

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — the three points from my last pass are genuinely addressed, and the shape of the fix is better than what I asked for.

  • Worker authorization. jwt_verifier is the right call. Modelling it on hostname_callable fits the project's style, and returning a mapping rather than a bool leaves room to carry more than authorized later. The fail-closed path — trusted_jwks_url set with neither jwt_issuer nor jwt_verifier raises instead of silently accepting anything the JWKS signed — covers the worst configuration, and test_configured_verifier_rejects_unauthorized_identity is exactly the specification test I was after.
  • Audience. The cast is gone, the core annotation now reads str | Sequence[str] | None with a comment explaining why None is load-bearing, and the provider.yaml wording matches what PyJWT actually does.
  • Config reference. Both security-relevant consequences now sit on the options a Deployment Manager reads first.
  • Single source of truth for the mode. _oidc_enabled() cached, with a docstring saying why, and the fixture clears it.

On the point you answered rather than implemented — jwt_issuer set with no jwt_verifier still accepting any identity from that issuer — I'm satisfied with the resolution. It's opt-in, and trusted_jwks_url's help says so in as many words. Documenting the trade-off where someone decides whether to enable it is the right trade for a provider-level feature.

Three nits inline, none blocking. The first is the one I'd most like to see land — it's small, and the rest of the codebase already does it the other way.

One cosmetic thing outside the diff: the PR description still calls the option token_verifier; it's jwt_verifier.


This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@robertpofuk
robertpofuk force-pushed the edge3-oidc-worker-token-auth branch 2 times, most recently from 1102a73 to 67ab2a6 Compare September 15, 2026 13:30
@robertpofuk

Copy link
Copy Markdown
Contributor Author

Added 2 new types: _OidcAuthenticator and WorkerTokenContext following same logic. This will add for now "method" as param but we can in future add more info theoretically neede (i.e. queryParams or whatever and extend), so both intput and oput are externdable.

Made validator async so should also be future proof.

@robertpofuk

Copy link
Copy Markdown
Contributor Author

Added it as new commit so changes are clearly visible

@robertpofuk
robertpofuk requested a review from potiuk September 15, 2026 16:11
Edge workers could only authenticate with tokens signed by the shared
[api_auth] jwt_secret, forcing every deployment to distribute that secret
to each worker. Deployments that already run a central OpenID Connect
provider want their workers to authenticate with provider-minted tokens
instead, so the secret never has to leave the API server.

Because an externally issued token proves only who signed it, a trusted
signature alone is not authorization to act as a worker: when issuer
verification is skipped the configuration is rejected unless an explicit
jwt_verifier decides which identities are allowed. Verification stays
opt-in behind [edge] trusted_jwks_url, leaving the shared-secret path
untouched for existing deployments.
…rifier

Edge worker token validation ran through the synchronous validated_claims
path, which does network I/O across per-request event loops once a JWKS URL
is configured. Switch the authorization entry points to await the async
validator like the rest of the codebase, resolve the jwt_verifier import once
when the OIDC authenticator is built so a bad path fails fast, and hand the
verifier a WorkerTokenContext carrying the request method so the input can
grow without a breaking signature change while jwt_verifier is unreleased.
@ashb
ashb force-pushed the edge3-oidc-worker-token-auth branch from 67ab2a6 to 70d233c Compare September 24, 2026 10:38
@ashb

ashb commented Sep 24, 2026

Copy link
Copy Markdown
Member

I've done a final rebase to re-run tests against latest main then we'll merge.

@ashb
ashb merged commit 0aa1b9a into apache:main Sep 24, 2026
80 checks passed
@boring-cyborg

boring-cyborg Bot commented Sep 24, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:edge Edge Executor / Worker (AIP-69) / edge3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants