Repository navigation
Add OIDC worker token verification to Edge3 - #72262
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
f4c978c to
94c638d
Compare
|
Tested on our internal environment by vendoring edge3 and re-building. Works nicely and makes our setup super simple since workers just order therir access via central IDP. |
0be8d57 to
203ac0c
Compare
|
Fixed CI failures and rebased on main. |
bc32bf5 to
7365323
Compare
36c8d1c to
a14db06
Compare
potiuk
left a comment
There was a problem hiding this comment.
Thanks for this — the shared-secret path is left genuinely untouched when trusted_jwks_url is empty, the config surface is well documented option-by-option, version_added: 4.4.0 is right off 4.3.1, and the tests are better than most: an autouse fixture clears the @cache between cases, and there is even an explicit test pinning the cache-stickiness behaviour. The [edge] section choice is correct too — the rest of edge3 reads from it.
This got two independent reviews: my own pass, and an adversarial second read. They converged on the same place, so I'm requesting changes rather than commenting.
To be clear about calibration first: nothing here breaks Airflow's documented security model. The feature is opt-in and a Deployment Manager has to enable it. What follows is about what the new code does once they do.
A trusted signature is not worker authorization
Details inline on auth.py:119. Today the signed method claim is the only thing that narrows what a worker token can do — a token minted for one endpoint cannot be replayed against another. OIDC mode drops that check and puts nothing in its place: no subject, no client, no scope restriction.
What is left is "this token was signed by a key in the configured JWKS, and it has iat and exp". That proves who signed the token; it does not establish that the identity is allowed to act as an edge worker. If the issuer serves any other workload — the normal case for a corporate IdP — an unrelated identity holding an aud-less token can register workers, fetch queued jobs, and change job states, since this function is the sole authentication dependency on those routes. That is strictly more access than a holder of the current shared secret has.
It compounds with the issuer default: [edge] jwt_issuer has no default, so the documented minimal configuration (trusted_jwks_url and nothing else) passes issuer=None, and PyJWT's _validate_iss returns immediately on None without checking anything. Enabling the feature in its simplest form therefore binds neither the issuer nor the identity.
What I'd like to see: an explicit worker-authorization policy driven by validated identity or scope claims, jwt_issuer required (failing closed at validator construction) whenever trusted_jwks_url is set, and an Edge-specific audience where the provider supports one.
The audience option documents behaviour the code does not have
Details inline on auth.py:81-82. jwt_audience empty means audience=None, and JWTValidator leaves PyJWT's verify_aud enabled. PyJWT's _validate_aud returns early only when the token has no aud; a token that carries one raises InvalidAudienceError.
So the config help — "leave empty to skip audience verification" — describes the opposite of what happens. An operator who follows it with an ordinary IdP token gets 403s and no obvious reason why. Either require a configured audience for tokens that carry aud and reword the option to say that empty accepts only audience-less tokens, or implement genuine skipping explicitly.
The tests cannot see either of the above
Both findings live in how a signed token is validated, and no test validates one — the suite asserts validator attributes (validator.jwks is not None, validator.algorithm == ["RS512"]) and calls _check_method_claim with hand-built dicts. That is real coverage of the method-claim branch, and I don't want to undersell it, but it cannot catch an audience rejection or an unauthorized-identity acceptance.
Signed-token tests would close this: one proving a token from the trusted issuer belonging to an unrelated identity is rejected, and a pair covering tokens with and without aud under both an empty and a configured jwt_audience.
Smaller observations
auth.py:119— "is OIDC enabled?" is decided in two places with different lifetimes.jwt_validator()is@cached so the mode is fixed at first use, while_check_method_claimre-reads config on every request. They can disagree: a validator cached in shared-secret mode plus a config that now reports a JWKS URL gives shared-secret validation with the method check disabled — a shared-secret token valid for every method. Narrow, but this is the auth path, and one source of truth is cheap.provider.yaml:123— the config reference never mentions that enablingtrusted_jwks_urlremoves per-method token scoping, andjwt_issuer's description does not say that leaving it unset means the issuer is not checked. Those are the two security-relevant consequences and they currently live only in the PR description.- The PR description documents
oidc_jwks_url/oidc_issuer/oidc_audience/oidc_algorithms, but after the rename the keys aretrusted_jwks_url/jwt_issuer/jwt_audience/jwt_algorithm. Provider changelogs are regenerated from git history, so the stale names can reach release notes. - Dropping
nbffromrequired_claimsrelative to the core default is sensible for IdP-issued tokens — noting it only so the choice is on the record.
This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. After you've
addressed the points above and pushed an update, an Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
a14db06 to
f9bbd44
Compare
6cca294 to
0d37eb3
Compare
|
Adjusted implementation to ensure that at least some assertions are required for checking weather token is allowed to be used with oidc enabled. jwt_verifier concept is inspured by [core].hostname_callable so should fit overall style jwt_verifier is on purpose built to return dict back. This way future requirements can be added. For exmaple core Airflow hasfunction to get_teams (i think in executor impelemtations). So We can use this return as place to extend and return from jwt_verifier {authorized: true teams [x, y]}. Much cleaner to have compatible extension if we return dict then fixed values. |
0625cd1 to
f8ae22a
Compare
potiuk
left a comment
There was a problem hiding this comment.
Approving — the three points from my last pass are genuinely addressed, and the shape of the fix is better than what I asked for.
- Worker authorization.
jwt_verifieris the right call. Modelling it onhostname_callablefits the project's style, and returning a mapping rather than a bool leaves room to carry more thanauthorizedlater. The fail-closed path —trusted_jwks_urlset with neitherjwt_issuernorjwt_verifierraises instead of silently accepting anything the JWKS signed — covers the worst configuration, andtest_configured_verifier_rejects_unauthorized_identityis exactly the specification test I was after. - Audience. The
castis gone, the core annotation now readsstr | Sequence[str] | Nonewith a comment explaining whyNoneis load-bearing, and theprovider.yamlwording matches what PyJWT actually does. - Config reference. Both security-relevant consequences now sit on the options a Deployment Manager reads first.
- Single source of truth for the mode.
_oidc_enabled()cached, with a docstring saying why, and the fixture clears it.
On the point you answered rather than implemented — jwt_issuer set with no jwt_verifier still accepting any identity from that issuer — I'm satisfied with the resolution. It's opt-in, and trusted_jwks_url's help says so in as many words. Documenting the trade-off where someone decides whether to enable it is the right trade for a provider-level feature.
Three nits inline, none blocking. The first is the one I'd most like to see land — it's small, and the rest of the codebase already does it the other way.
One cosmetic thing outside the diff: the PR description still calls the option token_verifier; it's jwt_verifier.
This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
1102a73 to
67ab2a6
Compare
|
Added 2 new types: _OidcAuthenticator and WorkerTokenContext following same logic. This will add for now "method" as param but we can in future add more info theoretically neede (i.e. queryParams or whatever and extend), so both intput and oput are externdable. Made validator async so should also be future proof. |
|
Added it as new commit so changes are clearly visible |
Edge workers could only authenticate with tokens signed by the shared [api_auth] jwt_secret, forcing every deployment to distribute that secret to each worker. Deployments that already run a central OpenID Connect provider want their workers to authenticate with provider-minted tokens instead, so the secret never has to leave the API server. Because an externally issued token proves only who signed it, a trusted signature alone is not authorization to act as a worker: when issuer verification is skipped the configuration is rejected unless an explicit jwt_verifier decides which identities are allowed. Verification stays opt-in behind [edge] trusted_jwks_url, leaving the shared-secret path untouched for existing deployments.
…rifier Edge worker token validation ran through the synchronous validated_claims path, which does network I/O across per-request event loops once a JWKS URL is configured. Switch the authorization entry points to await the async validator like the rest of the codebase, resolve the jwt_verifier import once when the OIDC authenticator is built so a bad path fails fast, and hand the verifier a WorkerTokenContext carrying the request method so the input can grow without a breaking signature change while jwt_verifier is unreleased.
67ab2a6 to
70d233c
Compare
|
I've done a final rebase to re-run tests against latest main then we'll merge. |
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Why
The Edge worker API currently only verifies worker tokens signed with the shared
[api_auth] jwt_secret(symmetric HS512). This adds optional verification against a trusted OpenID Connect provider, so edge workers can authenticate with tokens minted by an external identity provider instead of the shared secret.What
When
[edge] trusted_jwks_urlis set, worker tokens are verified against the provider JWKS using asymmetric signatures, checking theissand (optionally)audclaims, and then an authorization hook decides whether the identity may act as an edge worker. New[edge]options:trusted_jwks_url— provider JWKS URL; enables OIDC verification when set. Enabling it disables the per-request signedmethodclaim check (externally issued tokens do not carry it), so authorization is delegated tojwt_verifier.jwt_issuer— expectedissclaim. Leave empty to skip issuer verification; set it to bind tokens to your provider.jwt_audience— expectedaudclaim. When set, the token must carry a matchingaud; when empty, only tokens without anaudclaim are accepted (a token that carries anaudis rejected).jwt_algorithm— accepted signing algorithms, comma-separated (defaultRS256).jwt_leeway— clock-skew leeway in seconds for OIDC tokens (default30).jwt_verifier— import path of a callableverify(claims: dict, context: WorkerTokenContext) -> {"authorized": bool}run after signature/issuer/audience checks. It receives the validated claims and a request context carrying the requestmethod. A falsy result or a raised exception rejects the request. Use it to restrict which identities (e.g. bysub,client_id, orscope) may act as edge workers. When empty, any validly signed token from the trusted issuer is authorized.When
trusted_jwks_urlis empty (the default), the existing shared-secret verification is used unchanged, so existing deployments are unaffected.Uses the existing core
JWKS/JWTValidator. The coreJWTValidator.audienceannotation is widened tostr | Sequence[str] | None(the runtime already acceptsNone) so an empty edge audience can be passed honestly instead of via a cast.