Repository navigation
Fix airflowctl trigger sending unrecognized fields to older API servers - #70328
namita-can wants to merge 1 commit into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
potiuk
left a comment
There was a problem hiding this comment.
Thanks — the problem is real: newer optional fields like partition_key and bundle_version being serialised as null will trip extra="forbid" on an older API server. Worth fixing.
But exclude_none=True overshoots, and I think it breaks the current server while fixing the old one. In airflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_run.py:226 the server-side model declares:
logical_date: AwareDatetime | None— no default, so in Pydantic that is a required field that must be present in the body, even when null. Every sibling field has = None or a default_factory; logical_date deliberately does not, and the @model_validator doesn't supply one either.
So triggering a Dag run without an explicit logical date — the normal case — will now omit logical_date entirely and get a 422 "Field required" from a current server. That's a worse regression than the one being fixed.
Details and a suggested direction inline.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
DagOperations.trigger() didn't exclude None fields when serializing the request, unlike create_event() in the same file. This meant every trigger request included partition_key and bundle_version (added in Airflow 3.2.0), which older API servers reject as unrecognized fields. Adds exclude_none=True to match create_event()'s existing pattern, plus a regression test asserting unset fields are excluded from the request body. Narrow Trigger field exclusion to partition_key/bundle_version
376700a to
bce0ee9
Compare
potiuk
left a comment
There was a problem hiding this comment.
Thanks — the logical_date half is fixed correctly, and the comment explaining why it has to stay in the body (no default, so required-but-nullable) is exactly the kind of note that stops someone re-introducing exclude_none later. The test asserting its presence is right too.
Two things still open, one of them new.
The unconditional exclusion silently drops user-supplied values
I checked whether this is reachable, and it is. TriggerDAGRunPostBody declares both fields for real, and cli_config.py:714 generates a --<field> flag for every model field except schema_ (excluded_parameters = ["schema_"]). So airflowctl dags trigger --partition-key foo is a valid command today: argparse accepts it, the model carries the value, and trigger() now strips it before sending. The Dag run is created without the partition key and nothing reports an error.
That is a worse failure than the one being fixed. A 422 from an old server is loud and diagnosable; this is a user watching their partition key disappear with a successful exit code.
Detail and a suggested fix inline.
The round-trip test
This was in my last review and is worth restating: httpx.MockTransport returns a canned 200 without validating the body, so no assertion about dict keys can catch "the server would have rejected this". Validating the captured body through the server's own model would have caught both the original logical_date bug and this one:
from airflow.api_fastapi.core_api.datamodels.dag_run import TriggerDAGRunPostBody as ServerBody
ServerBody.model_validate(captured_body)Worth adding whichever way the exclusion goes.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
| self.response = self.client.post( | ||
| f"dags/{dag_id}/dagRuns", json=trigger_dag_run.model_dump(mode="json") | ||
| f"dags/{dag_id}/dagRuns", | ||
| json=trigger_dag_run.model_dump(mode="json", exclude={"partition_key", "bundle_version"}), |
There was a problem hiding this comment.
Excluding these unconditionally means they can never be sent, including to servers that support them — and both are user-settable today (cli_config.py:714 generates a CLI flag for every model field bar schema_), so --partition-key foo is silently discarded rather than rejected.
Conditioning the exclusion on the value keeps both sides working:
| json=trigger_dag_run.model_dump(mode="json", exclude={"partition_key", "bundle_version"}), | |
| exclude = { | |
| field | |
| for field in ("partition_key", "bundle_version") | |
| if getattr(trigger_dag_run, field) is None | |
| } | |
| self.response = self.client.post( | |
| f"dags/{dag_id}/dagRuns", | |
| json=trigger_dag_run.model_dump(mode="json", exclude=exclude), | |
| ) |
An old server no longer sees the unknown null fields; a new server still receives values the user actually set. A user who sets --partition-key against an old server then gets a 422, which is the correct outcome — that server genuinely cannot honour the request, and saying so is better than pretending it worked.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
|
Closing as already covered by #73763 |
DagOperations.trigger() didn't exclude None fields when serializing the request, unlike create_event() in the same file. This meant every trigger request included partition_key and bundle_version (added in Airflow 3.2.0), which older API servers reject as unrecognized fields.
Adds exclude_none=True to match create_event()'s existing pattern, plus a regression test asserting unset fields are excluded from the request body.
Manually verified against a live Airflow 3.1.x deployment (Charmed Airflow on Kubernetes): confirmed dags trigger fails with the extra_forbidden error on the unpatched code, and succeeds cleanly — both with and without --logical-date passed explicitly, once the fix is applied.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.