Skip to content

Rescheduled sensors can hang before poke while fetching first reschedule start date #68010

Closed

Description

@hkc-8010

A rescheduled sensor task can get stuck before entering the sensor poke() method when the worker restarts an attempt that already has TaskReschedule rows.

Observed behavior:

  • The task attempt starts and emits task/listener startup logs.
  • The sensor never emits its normal ExternalTaskSensor "Poking for Dag ..." log line.
  • No external Dag run count request is made for that stuck restart.
  • The task remains running until it is killed.
  • A retry of the same task instance reaches poke() immediately and succeeds.

The source path that matches this behavior is BaseSensorOperator.execute() calling RuntimeTaskInstance.get_first_reschedule_date() before poke(). When task_reschedule_count > 0, the Task SDK currently asks the supervisor to fetch the first TaskReschedule.start_date, which adds a pre-poke supervisor/API round trip for rescheduled sensors.

Expected behavior:
A rescheduled sensor restart should not need an extra pre-poke supervisor/API request for metadata the API server already knows when it creates the task run context. The worker should receive the first reschedule start date in TIRunContext and use it directly, while preserving the existing supervisor request as a compatibility fallback.

Proposed fix:
Add first_task_reschedule_start_date to TIRunContext, populate it in the Execution API run response when the task instance has reschedules, and have the Task SDK use that value before falling back to GetTaskRescheduleStartDate.

This avoids the pre-poke blocking point and keeps older API/server combinations compatible.

Activity

  1. added a commit that references this issue on Jun 12, 2026
    f3bd4aa
  2. added a commit that references this issue on Jul 3, 2026
    49a6aa7
  3. added a commit that references this issue on Aug 5, 2026
    0988540
  4. added a commit that references this issue on Aug 23, 2026
    3f471bc
  5. added a commit that references this issue on Sep 10, 2026
    7209bb3
  6. added a commit that references this issue on Sep 24, 2026
    fbbc067
  7. added a commit that references this issue on Oct 4, 2026
    65a9a8c
  8. added a commit that references this issue on Oct 5, 2026
    53228fc
  9. added 2 commits that reference this issue on Oct 10, 2026
    37061b5
    a88e6b1
  10. added a commit that references this issue on Oct 12, 2026
    3af85e2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions