Repository navigation
Allow task to depend on specific tasks from previous dag_run #60328
Description
Description
depends_on_past=True currently allows a task to depend only on the same task in the previous dag_run.
There is no native way for a task to depend on the successful completion of multiple specific tasks from the previous dag_run (within the same DAG). Achieving this today requires custom sensors that directly query Airflow metadata tables.
Use case/motivation
In stateful or partially retryable pipelines, it is common to require that a group of tasks in the previous dag_run has completed successfully before continuing execution in the next run.
Example:
A → B → C → D → E
If:
dag_run_1 fails at task D
Then for dag_run_2: Task C should not start until C, D, and E from dag_run_1 have all succeeded
Today:
depends_on_past is insufficient (checks only the same task)
trigger_rule applies only within the same dag_run
Users must implement custom PythonSensors that query DagRun / TaskInstance
This adds boilerplate, DB coupling, and operational complexity for a fairly common dependency pattern.
What would you like to happen?
Provide a declarative, first-class way for a task to depend on specific tasks from the previous dag_run.
Illustrative example (API shape open for discussion):
C = PythonOperator(
task_id="C",
depends_on_past=True,
depends_on_previous_task_ids=["C", "D", "E"],
)
Semantics:
In dag_run(N), task C is scheduled only if the listed tasks in dag_run(N-1) all finished with success
First dag_run is not blocked
Backward compatible and fully opt-in
This would remove the need for custom sensors while keeping cross-run dependencies explicit and readable.
Related issues
No known existing issue covering this specific capability.
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jan 9, 2026
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
abhijeets25012-tech commentedon Jan 9, 2026
i would like to contribute
SameerMesiah97 commentedon Jan 9, 2026
You are free to open a PR but I would advise you to review the existing PR once it is done or pick another issue you feel you can handle.
Parthsuii commentedon Jan 10, 2026
Hi I’d like to work on this
shashbha14 commentedon Jan 12, 2026
I would like to work on this issue.
yuseok89 commentedon Aug 18, 2026
Before adding a new parameter, I wanted to check whether the same behaviour can be had with what's already there, and see whether it covers your case.
In the example from the issue, C needs
{C, D, E}, which is exactly "C itself plus everything transitively downstream of C".wait_for_downstream=Truedoes almost that. It turns ondepends_on_pastas well and then additionally waits on downstream tasks, but only immediately downstream ones. So setting it on C covers{C, D}and misses E.downstream_task_idsis just a set, and adding a transitive edge doesn't change execution order (the chain already enforces it), so a single edge closes that gap.wait_for_downstreamnow covers{D, E}and the implieddepends_on_pastcoversC, giving{C, D, E}. The first Dag run isn't blocked either, since the check passes when there's no previous run.Would this work in your actual pipeline, or does it miss something? If it covers the case, documenting the pattern under
wait_for_downstreammight be more useful than a new argument. If it doesn't, knowing why would sharpen the case for a first-class parameter.mahesh-desu commentedon Sep 25, 2026
I'd like to work on this - plan to check wait_for_downstream coverage first, then propose prev_task_ids semantics building on #64653 (batch query, parse-time validation, XCom path). May I proceed?