Skip to content

Let listeners see the end date of a skipped task instance - #75685

Merged
ashb merged 1 commit into
apache:mainfrom
kacpermuda:sdk-set-end-date-on-skip
Oct 27, 2026
Merged

Let listeners see the end date of a skipped task instance#75685
ashb merged 1 commit into
apache:mainfrom
kacpermuda:sdk-set-end-date-on-skip

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Listeners are notified with the runtime task instance after the task runner has reported its terminal state, so they can read end_date from it, the same value that is stored for the task instance. That works for success, failure and retry, where the task runner sets ti.end_date. But a task that raised AirflowSkipException, or a TriggerDagRunOperator task skipped or failed because the Dag run it triggers already exists, only put the end date in the message to the supervisor and left ti.end_date unset. A listener then has no way to get the end date of that task instance.

A listener can't take the time itself either: finalize() notifies listeners only after the state change callbacks, extra links and rendered fields are handled, and those failed/skipped states reach the API server only after the task process exits.

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

🤖 Generated with Claude Code

@kacpermuda @claude
Let listeners see the end date of a skipped task instance
e92fdb5 
Listeners get the runtime task instance after its terminal state was
reported, but a task that skipped itself, or whose triggered Dag run already
existed, reported an end date without keeping it on the task instance. A
listener had no way to read the end date that was stored for it.

Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda marked this pull request as ready for review October 26, 2026 18:33
@ashb ashb added the backport-to-v3-4-test Backport to v3-4-test label Oct 27, 2026
@ashb
ashb merged commit bec92bc into apache:main Oct 27, 2026
126 of 138 checks passed
@ashb ashb added this to the Airflow 3.4.0 milestone Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

vatsrahul1001 pushed a commit that referenced this pull request Oct 27, 2026
@kacpermuda @vatsrahul1001
[v3-4-test] Let listeners see the end date of a skipped task instance (…
7b40cea 
…#75685)

(cherry picked from commit bec92bc)

Co-authored-by: Kacper Muda <mudakacper@gmail.com>
vatsrahul1001 pushed a commit that referenced this pull request Oct 27, 2026
@github-actions @kacpermuda
[v3-4-test] Let listeners see the end date of a skipped task instance (…
639d37d 
…#75685) (#75722)

(cherry picked from commit bec92bc)

Co-authored-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda
kacpermuda deleted the sdk-set-end-date-on-skip branch October 27, 2026 08:55
@kacpermuda kacpermuda mentioned this pull request Oct 27, 2026
Merged
1 task done
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
f1ee928 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
d314e91 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
316059b 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
@vatsrahul1001 vatsrahul1001 mentioned this pull request Oct 27, 2026
Closed
1 task done
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
c55902d 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
7ac372a 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Oct 27, 2026
@kacpermuda @claude
Note that Airflow 3.4 keeps the end date of skipped task instances
38df398 
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
@myps6415 myps6415 mentioned this pull request Oct 28, 2026
Open
4 tasks done
myps6415 added a commit to myps6415/airflow that referenced this pull request Oct 28, 2026
@myps6415
Add regression test for task.duration on skipped tasks
10b139d 
The task.duration metric went missing for skipped tasks because finalize()
only emits it when ti.end_date is set, and the skip path left it unset.
apache#75685 fixed the code path for listeners but did not assert on the metric,
so nothing guards against the gate being broken again.
mobuchowski pushed a commit that referenced this pull request Oct 28, 2026
@kacpermuda @claude
Use task instance end date as OpenLineage terminal event time (#75328)
ce75cee 
* Use the task instance end date as OpenLineage terminal event time

The Airflow 3 task runner sets the end date and reports it to the API
server before it notifies listeners, but the listener stamped complete,
fail and skipped events with the time of its own notification. That is
later than the stored end date by however long the Airflow callbacks took.

The task instance facet also never carried a duration on Airflow 3: the
runtime task instance has none, as the API server computes it after the
listener ran. It is now derived from the same end date as the event time.

Co-Authored-By: Claude <noreply@anthropic.com>

* Keep the stored task instance duration in the OpenLineage facet

Airflow 2 stores the duration before the listener is notified, while the
end date passed to the facet there is the notification time, taken after
the success callback ran. Computing the duration from it made the facet
disagree with the duration shown in the UI.

Co-Authored-By: Claude <noreply@anthropic.com>

* Note that Airflow 3.4 keeps the end date of skipped task instances

Since #75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>

* Include start and end dates in the OpenLineage task instance facet

Consumers had only the event time and a duration to work out when a task
instance ran, and the event time of a terminal event isn't always its end
date. The dates the task instance holds are the ones the UI shows.

Co-Authored-By: Claude <noreply@anthropic.com>

* Check task instance dates in the OpenLineage system tests

The facet now carries the start and end dates; locking them in on the
success, skip, failure and retry paths keeps a regression there from
going unnoticed.

Co-Authored-By: Claude <noreply@anthropic.com>

* Require a number for the duration in the OpenLineage system tests

A numeric string would pass the previous check, while the duration must be
emitted as a JSON number, as in the Airflow 2 expectations.

Co-Authored-By: Claude <noreply@anthropic.com>

* Keep OpenLineage task instance dates consistent with what Airflow stores

Address review: on Airflow 2 the event time and the facet's end date came
from different values, a retry reported the previous try's duration, and
before Airflow 3.2.2 a task resumed from deferral reported its resume time
as start date and a duration that left out the time before it deferred.

Co-Authored-By: Claude <noreply@anthropic.com>

* Check task instance dates per Airflow version in the OpenLineage system tests

A task resumed from deferral reports no start date and duration before
Airflow 3.2.2, and Airflow 2 events now carry the dates too, so the
expectations need to say which versions get what. That boundary is a
patch release, which the expected-event files couldn't express.

Co-Authored-By: Claude <noreply@anthropic.com>

* Expect no duration from a running Airflow 2 task instance in the OpenLineage test

Its stored duration is the previous try's, which the facet no longer reports.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
@vatsrahul1001 vatsrahul1001 mentioned this pull request Oct 28, 2026
Closed
@hussein-awala hussein-awala mentioned this pull request Oct 29, 2026
Closed
7 of 36 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashb ashb ashb approved these changes
@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner
+1 more reviewer
@SameerMesiah97 SameerMesiah97 SameerMesiah97 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:task-sdk backport-to-v3-4-test Backport to v3-4-test

Projects

None yet

Milestone

Airflow 3.4.0

Development

Successfully merging this pull request may close these issues.

3 participants

@kacpermuda @ashb @SameerMesiah97