Expose task instance loop pass and expansion size to listeners by kacpermuda · Pull Request #75132 · apache/airflow

kacpermuda · 2026-10-19T18:00:27Z

Listener consumers such as OpenLineage should be able to trace an event back to the task execution that produced it from what the listener gets, not by looking the attempt up by ti.id in the metadata database. For mapped tasks and task loops part of that wasn't exposed:

LoopContext.region_id: the loop region holding the iteration. A mapped task's own region_id is its expansion's, so this is what ties it to the other tasks of the same pass.
LoopContext.trajectory, forked_from_region_id and resumes_from_index: which rerun of the loop an iteration ran in, and which execution it replaced from which iteration on. Before, ordering reruns meant comparing the UUIDv7 region ids of two events. With this new information, we know everything from a single event.
Public RuntimeTaskInstance.loop, loop_decision and expanded_ti_count. A gate's decision was only in the private _loop_decision, so its listener couldn't tell until being met from max_iterations being reached.
get_task_instance_execution_position(ti, session=...), so listeners called on the API server (manual state changes) get the same data as the worker, instead of reading dynamic_region rows.
TIRunContext.expanded_ti_count: the number of live instances in the mapped task's expansion, counted within the iteration inside a loop. Without it a listener doesn't know how many events to expect, so it can't tell "3 of 20 instances failed" or that all of them finished. This one was not required for backtracing, but it's a really useful piece of information.

The fields go into the unreleased 2026-10-30 Execution API version, so they can ship with 3.4.0.

related: #74752

Was generative AI tooling used to co-author this PR?

Yes (Claude Code)

Generated-by: Claude Code (Opus 5.5) following the guidelines

Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
When adding dependency, check compliance with the ASF 3rd Party License Policy.
For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

🤖 Generated with Claude Code

ashb

I'm not sure integer trajectory is right way to expose this though

ashb · 2026-10-19T18:46:20Z

I'll need to take another look from a computer

kacpermuda · 2026-10-20T10:13:27Z

I'm not sure integer trajectory is right way to expose this though

The trajectory itself is identified by the loop region UUID (loop.region_id). The integer is only that region’s position in the chain of forks, the same number the task log path already shows as pass=3.2. I’d like to keep it so a single event can tell which rerun it comes from, and whether it’s newer, without having seen the other region’s UUID (UUIDv7 ordering needs both). I reworded the docs and docstrings to make that split explicit. Happy to rename it if “trajectory” clashes with something different you have in mind.

ashb

Broadly approved, could of nits/questions about the generated schema and models

…ners A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. Co-Authored-By: Claude <noreply@anthropic.com>

Co-Authored-By: Claude <noreply@anthropic.com>

…ed attempts Co-Authored-By: Claude <noreply@anthropic.com>

Co-Authored-By: Claude <noreply@anthropic.com>

github-actions · 2026-10-21T11:32:38Z

Backport failed to create: v3-4-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-4-test

You can attempt to backport this manually by running:

cherry_picker e6a2c9f v3-4-test

This should apply the commit to the v3-4-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>

…eners (apache#75132) A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. (cherry picked from commit e6a2c9f) Co-authored-by: Claude <noreply@anthropic.com>

…eners (#75132) (#75309) A single event about a task instance should be enough to tell which execution it describes: which loop pass and rerun of the loop it ran in, how many instances its mapped expansion has, and why a loop gate stopped. Workers only had the loop iteration, a mapped task inside a loop could not be tied to the pass it ran in, reruns could only be ordered by comparing region ids from two events, and server side consumers such as listeners had to read region rows to place a stored task instance. (cherry picked from commit e6a2c9f) Co-authored-by: Claude <noreply@anthropic.com>

Uses what apache#75132 exposes, so a single event tells which rerun of a loop it comes from, whether that rerun superseded an earlier one, and how many mapped instances to expect, and events for manual state changes get the same details as worker events. Loop details are grouped in one object so it can describe enclosing loops if Airflow ever nests them, and iterations are counted from one as in Airflow. Co-Authored-By: Claude <noreply@anthropic.com>

…code The API server holds task instances without their task when a user sets a state, so those events never carried the airflow run facet the guide promised. The fallbacks for loop details only served Airflow builds from before apache#75132, which no release contains. Timing the stored position lookup makes its per task instance queries in the scheduler visible. Co-Authored-By: Claude <noreply@anthropic.com>