Pass Dag runs directly to Deadline references by imrichardwu · Pull Request #70714 · apache/airflow

imrichardwu · 2026-07-30T05:27:15Z

Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.

Pass the in-memory Dag run to Deadline references so built-in references avoid redundant database reads and custom references can evaluate the current state.

Deprecate required_kwargs while preserving it for existing custom references. Use the shared evaluator when recalculating queued-at deadlines.

closes: #70710

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

imrichardwu · 2026-07-30T05:37:30Z

@ferruzzi can I get a review?

github-actions · 2026-08-13T12:11:31Z

uv.lock on main just moved via #71520 ("Restore the evaluation extra of google-cloud-aiplatform"), commit b69253b and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

ferruzzi · 2026-08-25T21:45:03Z

Heads up on sequencing: this PR now overlaps two others in _recalculate_dagrun_queued_at_deadlines.

Fix TypeError clearing a DAG run when a queued-at deadline interval is stored as JSON #70370 (opened 24 July) is the targeted fix for the JSON-interval TypeError on that function. It is milestoned 3.3.2 and carries backport-to-v3-3-test.
Re-anchor custom queued deadline references when a Dag run is cleared #71850 (opened 19 August) persists the evaluation timing so custom queued-anchored references get re-anchored on clear. It also conflicts with this PR across sevreral files.

This PR fixes the same TypeError incidentally, and I want to be explicit that that does not make #70370 redundant. This is a refactor with a significant.rst newsfragment and a deprecation, so it is not backport material; #70370 is the only route the fix has to 3.3.x, where it currently affects every queued-at deadline. Please do not let this PR be used as a reason to close that one.

The cheapest merge order looks like it will be to merge #70370 first, then rebase here. #70370 adds decode_deadline_alert_model() and resolve_deadline_alert_interval() to serialization/decoders.py, which is exactly what this PR hand-rolls at taskinstance.py:255-260. Rebasing onto it lets you delete those lines.

ferruzzi · 2026-08-25T22:08:00Z

Adding a fourth PR to the picture since it breaks something here concretely.

#71802 adds a core-side SerializedVariableInterval, makes decode_deadline_alert return that instead of VariableInterval, and deletes resolve() from the SDK entirely.

That will break test_clear_task_instances_recalculates_variable_interval_deadline, which patches airflow.sdk.definitions.deadline.Variable.get. Worth knowing now rather than after a rebase.

Also worth noting, #71850's diff still has required_kwargs: set[str] = set() on
SerializedBaseDeadlineReference, which this PR removes, and it adds an evaluation_timing class attribute immediately below it. So be careful when rebasing, whichever lands first.

seanghaeli · 2026-10-13T17:01:20Z

Merging #68917 and then this one

seanghaeli · 2026-10-13T23:05:43Z

@ferruzzi or @imrichardwu , would one of you be able to rebase this?

Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

The custom deadline reference examples returned the dagrun attribute directly, which read as if the attribute was the required return value rather than an input available to the author's business logic. Type the evaluation dagrun with the shared DagRunProtocol so authors get a concrete contract for what is available, and show the attribute feeding into business logic in the examples.

A custom deadline reference whose _evaluate_with declares **kwargs is expected to receive the full evaluation context, matching the standard Python "accept anything" convention rather than having the context filtered away. Align the test with that behaviour.

Clarify the supported Dag run interface for deadline references and document the migration away from required_kwargs before Airflow 4.0.

Updated the DeadlineDagRunProtocol class to eliminate its inheritance from DagRunProtocol, simplifying the interface for deadline references. The class now directly defines the necessary attributes for evaluating deadlines without relying on the ORM DagRun structure.

…evaluations This update introduces warning logs in the SerializedReferenceModels for cases where a DagRun's logical date or queued at time is missing. Additionally, the test suite has been enhanced to verify that these warnings are correctly logged when the respective attributes are None.

…og messages

…adline evaluations

imrichardwu · 2026-10-14T17:39:28Z

@ferruzzi or @imrichardwu , would one of you be able to rebase this?

Done

seanghaeli · 2026-10-14T19:37:41Z

@ferruzzi looks good to merge

ferruzzi

Thanks for sticking with this one, I know it was a long wait.

…ence The recompute now asks the reference for the new time (apache#70714), so a reference whose evaluation returns something other than the bare queued_at is what separates re-anchoring through the reference from the built-in formula.

github-actions · 2026-10-18T08:58:38Z

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-4-test

You can attempt to backport this manually by running:

cherry_picker 637b741 v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

github-actions · 2026-10-18T10:29:03Z

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-4-test

…74986) * Fix custom Deadline references with optional keyword arguments Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly. (cherry picked from commit 637b741) Co-authored-by: Richard <richard9@ualberta.ca> Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

…ence The recompute now asks the reference for the new time (apache#70714), so a reference whose evaluation returns something other than the bare queued_at is what separates re-anchoring through the reference from the built-in formula.

imrichardwu requested review from XD-DENG, amoghrajesh, ashb, bolkedebruin and kaxil as code owners July 30, 2026 05:27

boring-cyborg Bot added area:dag-processor area:deadline-alerts AIP-86 (former AIP-57) area:task-sdk kind:documentation labels Jul 30, 2026

imrichardwu marked this pull request as draft July 30, 2026 05:29

imrichardwu force-pushed the fix/deadline-reference-context branch 2 times, most recently from e9c48fd to 3b99b93 Compare July 30, 2026 05:33

imrichardwu marked this pull request as ready for review July 30, 2026 05:33

imrichardwu force-pushed the fix/deadline-reference-context branch from 3b99b93 to db15212 Compare July 31, 2026 01:39

ferruzzi reviewed Aug 6, 2026

View reviewed changes

Comment thread airflow-core/docs/howto/deadline-alerts.rst Resolved

Comment thread airflow-core/src/airflow/models/deadline.py Resolved

Comment thread airflow-core/src/airflow/models/deadline.py Resolved

Comment thread airflow-core/src/airflow/models/deadline.py Resolved

imrichardwu requested a review from ferruzzi August 7, 2026 16:01

ferruzzi reviewed Aug 10, 2026

View reviewed changes

Comment thread task-sdk/src/airflow/sdk/definitions/deadline.py Resolved

Comment thread airflow-core/src/airflow/serialization/definitions/deadline.py Resolved

Comment thread airflow-core/docs/howto/deadline-alerts.rst Resolved

imrichardwu requested a review from ferruzzi August 12, 2026 03:11

ferruzzi mentioned this pull request Aug 18, 2026

A NULL column silently suppresses the deadline, and the warning blames a missing DagRun #71750

Closed

1 of 2 tasks

Prab-27 reviewed Aug 18, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/deadline.py Resolved

ferruzzi reviewed Aug 19, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/taskinstance.py Resolved

ferruzzi reviewed Aug 19, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/models/test_taskinstance.py Resolved

Comment thread airflow-core/tests/unit/models/test_taskinstance.py Resolved

imrichardwu requested a review from ferruzzi August 22, 2026 05:29

ferruzzi mentioned this pull request Aug 25, 2026

Fix TypeError clearing a DAG run when a queued-at deadline interval is stored as JSON #70370

Merged

3 tasks done

ferruzzi mentioned this pull request Aug 25, 2026

Re-anchor custom queued deadline references when a Dag run is cleared #71850

Open

1 task done

ferruzzi mentioned this pull request Aug 25, 2026

Add SerializedVariableInterval for deadline alerts #71802

Merged

imrichardwu and others added 11 commits October 13, 2026 17:27

Fix custom Deadline references with optional keyword arguments

f113467

Preserve the Dag run evaluation context for custom references that accept additional keyword arguments, so they continue to evaluate correctly.

Update airflow-core/docs/howto/deadline-alerts.rst

e9268c5

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

Update airflow-core/src/airflow/models/deadline.py

6589d2c

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

Update airflow-core/src/airflow/models/deadline.py

950a9c5

Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>

Improve deadline reference type safety

5de1436

Clarify the supported Dag run interface for deadline references and document the migration away from required_kwargs before Airflow 4.0.

Update test for calculated deadline references to assert structured l…

de1ea8d

…og messages

Remove warning logs for missing logical date and queued at time in de…

ab49ec2

…adline evaluations

imrichardwu force-pushed the fix/deadline-reference-context branch from 4010284 to ab49ec2 Compare October 13, 2026 23:29

imrichardwu requested a review from seanghaeli October 13, 2026 23:30

seanghaeli mentioned this pull request Oct 13, 2026

A deadline alert that undoes the scheduler's database changes can crash the scheduler or lose the Dag run #74677

Open

seanghaeli approved these changes Oct 14, 2026

View reviewed changes

ferruzzi approved these changes Oct 15, 2026

View reviewed changes

ferruzzi merged commit 637b741 into apache:main Oct 15, 2026
108 checks passed

imrichardwu deleted the fix/deadline-reference-context branch October 17, 2026 19:00

vatsrahul1001 added this to the Airflow 3.4.0 milestone Oct 18, 2026

vatsrahul1001 mentioned this pull request Oct 28, 2026

Status of testing of Apache Airflow 3.4.0rc3 #75865

Closed

Conversation

imrichardwu commented Jul 30, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

imrichardwu commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ferruzzi commented Aug 25, 2026

Uh oh!

ferruzzi commented Aug 25, 2026

Uh oh!

seanghaeli commented Oct 13, 2026

Uh oh!

seanghaeli commented Oct 13, 2026

Uh oh!

imrichardwu commented Oct 14, 2026

Uh oh!

seanghaeli commented Oct 14, 2026

Uh oh!

ferruzzi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Oct 18, 2026

Backport failed to create: v3-4-test. View the failure log Run details

Uh oh!

github-actions Bot commented Oct 18, 2026

Backport successfully created: v3-4-test

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented Aug 13, 2026 •

edited

Loading