Skip to content

Fix airflowctl trigger sending unrecognized fields to older API servers - #70328

Closed
namita-can wants to merge 1 commit into
apache:mainfrom
namita-can:fix/airflowctl-trigger-exclude-none
Closed

namita-can wants to merge 1 commit into
apache:mainfrom
namita-can:fix/airflowctl-trigger-exclude-none

Conversation

@namita-can

Copy link
Copy Markdown

DagOperations.trigger() didn't exclude None fields when serializing the request, unlike create_event() in the same file. This meant every trigger request included partition_key and bundle_version (added in Airflow 3.2.0), which older API servers reject as unrecognized fields.

Adds exclude_none=True to match create_event()'s existing pattern, plus a regression test asserting unset fields are excluded from the request body.

Manually verified against a live Airflow 3.1.x deployment (Charmed Airflow on Kubernetes): confirmed dags trigger fails with the extra_forbidden error on the unpatched code, and succeeds cleanly — both with and without --logical-date passed explicitly, once the fix is applied.


Was generative AI tooling used to co-author this PR?
  • [] Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg

boring-cyborg Bot commented Jul 23, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the problem is real: newer optional fields like partition_key and bundle_version being serialised as null will trip extra="forbid" on an older API server. Worth fixing.

But exclude_none=True overshoots, and I think it breaks the current server while fixing the old one. In airflow-core/src/airflow/api_fastapi/core_api/datamodels/dag_run.py:226 the server-side model declares:

logical_date: AwareDatetime | None

— no default, so in Pydantic that is a required field that must be present in the body, even when null. Every sibling field has = None or a default_factory; logical_date deliberately does not, and the @model_validator doesn't supply one either.

So triggering a Dag run without an explicit logical date — the normal case — will now omit logical_date entirely and get a 422 "Field required" from a current server. That's a worse regression than the one being fixed.

Details and a suggested direction inline.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

DagOperations.trigger() didn't exclude None fields when serializing
the request, unlike create_event() in the same file. This meant every
trigger request included partition_key and bundle_version (added in
Airflow 3.2.0), which older API servers reject as unrecognized fields.

Adds exclude_none=True to match create_event()'s existing pattern,
plus a regression test asserting unset fields are excluded from the
request body.

Narrow Trigger field exclusion to partition_key/bundle_version
@namita-can
namita-can force-pushed the fix/airflowctl-trigger-exclude-none branch from 376700a to bce0ee9 Compare August 13, 2026 23:07

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the logical_date half is fixed correctly, and the comment explaining why it has to stay in the body (no default, so required-but-nullable) is exactly the kind of note that stops someone re-introducing exclude_none later. The test asserting its presence is right too.

Two things still open, one of them new.

The unconditional exclusion silently drops user-supplied values

I checked whether this is reachable, and it is. TriggerDAGRunPostBody declares both fields for real, and cli_config.py:714 generates a --<field> flag for every model field except schema_ (excluded_parameters = ["schema_"]). So airflowctl dags trigger --partition-key foo is a valid command today: argparse accepts it, the model carries the value, and trigger() now strips it before sending. The Dag run is created without the partition key and nothing reports an error.

That is a worse failure than the one being fixed. A 422 from an old server is loud and diagnosable; this is a user watching their partition key disappear with a successful exit code.

Detail and a suggested fix inline.

The round-trip test

This was in my last review and is worth restating: httpx.MockTransport returns a canned 200 without validating the body, so no assertion about dict keys can catch "the server would have rejected this". Validating the captured body through the server's own model would have caught both the original logical_date bug and this one:

from airflow.api_fastapi.core_api.datamodels.dag_run import TriggerDAGRunPostBody as ServerBody
ServerBody.model_validate(captured_body)

Worth adding whichever way the exclusion goes.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

self.response = self.client.post(
f"dags/{dag_id}/dagRuns", json=trigger_dag_run.model_dump(mode="json")
f"dags/{dag_id}/dagRuns",
json=trigger_dag_run.model_dump(mode="json", exclude={"partition_key", "bundle_version"}),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excluding these unconditionally means they can never be sent, including to servers that support them — and both are user-settable today (cli_config.py:714 generates a CLI flag for every model field bar schema_), so --partition-key foo is silently discarded rather than rejected.

Conditioning the exclusion on the value keeps both sides working:

Suggested change
json=trigger_dag_run.model_dump(mode="json", exclude={"partition_key", "bundle_version"}),
exclude = {
field
for field in ("partition_key", "bundle_version")
if getattr(trigger_dag_run, field) is None
}
self.response = self.client.post(
f"dags/{dag_id}/dagRuns",
json=trigger_dag_run.model_dump(mode="json", exclude=exclude),
)

An old server no longer sees the unknown null fields; a new server still receives values the user actually set. A user who sets --partition-key against an old server then gets a 422, which is the correct outcome — that server genuinely cannot honour the request, and saying so is better than pretending it worked.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@shahar1

shahar1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Closing as already covered by #73763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Airflowctl dags trigger fails with extra_forbidden against Airflow < 3.2.0

3 participants