You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Task duration keeps increasing for skipped tasks due to DAG timeout #58536
Originally posted by Szjs-z July 29, 2025
Hi Airflow community,
I've encountered an issue where task duration keeps increasing for tasks that are skipped due to DAG timeout:
Problem Description:
When a DAG run times out due to dagrun_timeout, the unfinished tasks are marked as SKIPPED. However, in the Web UI, the task duration continues to increase even though the task state is SKIPPED :
Root Cause Analysis:
From code inspection in airflow/jobs/scheduler_job.py, the _schedule_dag_run method handles DAG timeout by:
if (
dag_run.start_dateanddag.dagrun_timeoutanddag_run.start_date<timezone.utcnow() -dag.dagrun_timeout
):
dag_run.set_state(DagRunState.FAILED)
unfinished_task_instances=session.scalars(
select(TI)
.where(TI.dag_id==dag_run.dag_id)
.where(TI.run_id==dag_run.run_id)
.where(TI.state.in_(State.unfinished))
)
fortask_instanceinunfinished_task_instances:
task_instance.state=TaskInstanceState.SKIPPEDsession.merge(task_instance)
session.flush()
self.log.info("Run %s of %s has timed-out", dag_run.run_id, dag_run.dag_id)
Setting DAG run state to FAILED
Marking unfinished tasks as SKIPPED
But it doesn't set end_date for these task instances
I'm wondering if this is a known issue or if there's a recommended way to handle this.
I have also been able to reproduce this issue on Airflow 2.11. I agree with Chris' analysis on the root cause.
The set_state method should be called for the unfinished task instances as well instead of directly setting the state in the DBso that the end_date is set.
The task instance duration should also be set.
Additionally, callbacks should be run because the task instance could have an on_skipped_callback.
This code only skips the running task instances. Downstream task instances remain in the None state. Why aren't all task instances skipped?
Activity
karenbraganz commentedon Nov 20, 2025
I have also been able to reproduce this issue on Airflow 2.11. I agree with Chris' analysis on the root cause.
set_statemethod should be called for the unfinished task instances as well instead of directly setting the state in the DBso that the end_date is set.tirkarthi commentedon Nov 22, 2025
See also #30264
steveahnahn commentedon Nov 22, 2025
Opened a PR to address this issue in #58588
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout
fix(scheduler): set end_date on tasks skipped by dagrun timeout