Skip to content

Task duration keeps increasing for skipped tasks due to DAG timeout #58536

Closed

Description

@karenbraganz

Discussed in #53868

Originally posted by Szjs-z July 29, 2025
Hi Airflow community,

I've encountered an issue where task duration keeps increasing for tasks that are skipped due to DAG timeout:

Problem Description:
When a DAG run times out due to dagrun_timeout, the unfinished tasks are marked as SKIPPED. However, in the Web UI, the task duration continues to increase even though the task state is SKIPPED :

Screenshot 2025-07-29 at 17 42 11

Root Cause Analysis:
From code inspection in airflow/jobs/scheduler_job.py, the _schedule_dag_run method handles DAG timeout by:

        if (
            dag_run.start_date
            and dag.dagrun_timeout
            and dag_run.start_date < timezone.utcnow() - dag.dagrun_timeout
        ):
            dag_run.set_state(DagRunState.FAILED)
            unfinished_task_instances = session.scalars(
                select(TI)
                .where(TI.dag_id == dag_run.dag_id)
                .where(TI.run_id == dag_run.run_id)
                .where(TI.state.in_(State.unfinished))
            )
            for task_instance in unfinished_task_instances:
                task_instance.state = TaskInstanceState.SKIPPED
                session.merge(task_instance)
            session.flush()
            self.log.info("Run %s of %s has timed-out", dag_run.run_id, dag_run.dag_id)
  1. Setting DAG run state to FAILED
  2. Marking unfinished tasks as SKIPPED
  3. But it doesn't set end_date for these task instances

I'm wondering if this is a known issue or if there's a recommended way to handle this.

Thanks,
Chris Zhao

Activity

  1. karenbraganz commented on Nov 20, 2025

    @karenbraganz
    CollaboratorAuthor

    I have also been able to reproduce this issue on Airflow 2.11. I agree with Chris' analysis on the root cause.

    • The set_state method should be called for the unfinished task instances as well instead of directly setting the state in the DBso that the end_date is set.
    • The task instance duration should also be set.
    • Additionally, callbacks should be run because the task instance could have an on_skipped_callback.
    • This code only skips the running task instances. Downstream task instances remain in the None state. Why aren't all task instances skipped?
  2. tirkarthi commented on Nov 22, 2025

    @tirkarthi
    Contributor

    See also #30264

  3. steveahnahn commented on Nov 22, 2025

    @steveahnahn
    Contributor

    Opened a PR to address this issue in #58588

  4. added 7 commits that reference this issue on Mar 10, 2026
    375dc64
    072041d
    36eefc3
    2b79864
    64899cc
    06edc79
    6ebc21a
  5. added 14 commits that reference this issue on Apr 5, 2026
    d1dc2ca
    7086ef5
    0c91eb2
    4061fe7
    24902f4
    c497cda
    c30beb5
    ff86be7
    9e738f5
    d817d2a
    e898899
    74c3ea4
    df2246a
    8d5248c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions