Skip to content

Fix MSGraphSensor timeout being reset on every re-defer - #75774

Merged
dabla merged 2 commits into
apache:mainfrom
stegololz:fix-msgraph-sensor-timeout
Oct 28, 2026
Merged

dabla merged 2 commits into
apache:mainfrom
stegololz:fix-msgraph-sensor-timeout

Conversation

@stegololz

@stegololz stegololz commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Fix MSGraphSensor timeout being reset on every re-defer

What

MSGraphSensor could never time out. It now raises AirflowSensorTimeout once timeout
seconds have passed since the first execute() of the try, so soft_fail / never_fail
work again. It also stops passing the sensor timeout to MSGraphTrigger, which used it as
the HTTP request timeout.

Why

A defer timeout is stored as utcnow() + timeout (AF3 execution API ti_update_state,
AF2 TaskInstance._defer_task), and timeout=None clears any earlier deadline. The sensor
alternated between an MSGraphTrigger defer without a timeout (clearing the deadline) and a
TimeDeltaTrigger sleep with the full timeout (restarting it), so the deadline moved forward
on every poll and was never reached.

Separately, timeout=self.timeout was passed to MSGraphTrigger, where it became the httpx
per-request timeout (7 days by default).

How

  • The first execute() records its start time and carries it in next_kwargs
    (sensor_started_at) through execute_complete and retry_execute; if it is missing
    (TIs deferred before upgrading), it falls back to now.
  • Each request defer gets the remaining time as its defer timeout, so a hung Graph call is
    bounded as well.
  • The deadline is checked after a not-done poll, in the same order as
    BaseSensorOperator.execute. Past the deadline, AirflowSensorTimeout is raised from the
    resumed method, so core resume_execution still applies soft_fail (and never_fail on
    3.1+). It is not retried.
  • The sleep between polls is capped at the remaining time and is always followed by one last
    poll, which is given at least retry_delay so that small clock differences between workers
    cannot cut it short. The sensor can therefore overrun its deadline by at most retry_delay.
    A done response that arrives after the deadline still succeeds.
  • The sleep defer keeps a deadline of the remaining time plus retry_delay, so a lost sleep
    trigger still ends in AirflowSensorTimeout.
  • The timeout is counted per try, like AF3 core sensors.
  • retry_execute now defers directly instead of calling execute().

The HTTP timeout stays at the trigger/hook default (None); each request is still bounded by
the remaining sensor time. A dedicated HTTP-timeout parameter can follow if needed.

How tested

  • New time_machine unit tests: first defer timeout and start time (trigger HTTP timeout is
    None), deadline not moving across re-defers, sleep capped at the remaining time,
    sleep defer deadline, AirflowSensorTimeout past the deadline, one last poll (given at least retry_delay) when
    resuming near, at or past the deadline, done response after the deadline,
    resume_execution with no flag / soft_fail / never_fail, missing-start-time fallback.
  • Existing end-to-end deferrable tests unchanged.
  • Whole azure sensors suite + msgraph trigger tests: 79 passed. With the source change
    reverted, the new sensor tests fail.
  • prek run --files <changed files> and mypy-providers pass.

🤖 Generated with Claude Code

A defer timeout becomes an absolute deadline of now + timeout, and
MSGraphSensor re-deferred with the full sensor timeout between polls
(while the request defer carried none), so the sensor never timed out.

Carry the first execute's start time in next_kwargs, give each request
defer the remaining time, and raise AirflowSensorTimeout after a
not-done poll once the timeout has elapsed, so soft_fail and never_fail
apply as for core sensors.

Stop passing the sensor timeout to MSGraphTrigger, where it was used as
the HTTP request timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@dabla dabla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related: #62241 (closed #62157 with the sliding re-defer timeout this PR replaces); PowerBI's PowerBITrigger enforces its timeout inside the trigger, so it was never affected.

Addressed

Nothing since the previous round: no reviews or threads on the PR yet.

Open

  • The premise of the PR holds, and it is the issue you remember. #62241 (March 2026, "Pass timeout to defer() in MSGraphSensor", closing #62157) put timeout=self.timeout on the sleep re-defer only. Core stores a deferral deadline as utcnow() + timeout on every defer and clears it on a defer without one (execution API ti_update_state, TaskInstance._defer_task on 2.x), so that fix restarted the clock on every poll and the MSGraphTrigger defer in between wiped it: the sensor still never timed out. PowerBI is a different mechanism: PowerBITrigger polls inside the trigger with a time.monotonic() loop bounded by its timeout, so its deadline never depended on re-defers and was not touched by #62241 or by #74675. This PR is the first fix of the MSGraph sensor itself.
  • [1] warning: the sleep defer now has no deadline at all; keep one above the sleep length.
  • [2] nit: timeout is None branches and their test are unreachable through the base class.
  • [3] nit: _anchor / started_at / sensor_started_at name one concept.
  • Not commented: tests_common deferrable_operator drops task.kwargs when it resumes, so the three unchanged end-to-end tests exercise the "missing start time" fallback, not the carried sensor_started_at; the kwargs round trip is covered by test_timeout_via_resume_execution and the direct unit tests, so no gap in coverage, only a harness limitation worth knowing. Pre-existing and out of scope: execute_complete returns None (success) when the event carries an empty response.

Head 3574742 reviewed in diff mode. Verified against the Task SDK and 2.11: BaseSensorOperator.resume_execution catches AirflowException (so AirflowSensorTimeout raised from the resumed method honours soft_fail on both, never_fail on 3.1+), self.timeout is always a number, AirflowSensorTimeout and timezone are in common.compat.sdk since before the 1.23.0 floor, datetime.UTC is fine with the provider's requires-python >= 3.11, time-machine is a devel-common dependency. Audits run: parity with BaseSensorOperator.execute (poke, timeout check, sleep, same message), lifecycle and timeout (per-defer deadlines on both deferral paths, which produced [1]), repeated side effects (retry_execute no longer re-enters execute), SDK subclass contract (produced [2]), docs and test wording, python checklist (vocabulary, [3]). No tests run locally; the PR body reports 80 passed on the azure sensors and msgraph trigger suites.

Give the TimeDeltaTrigger sleep a defer timeout of the remaining time
plus retry_delay, so a lost sleep trigger still ends in a sensor
timeout. Drop the unreachable timeout=None branches and use started_at
consistently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stegololz

Copy link
Copy Markdown
Contributor Author

Thanks for the review @dabla, all three points addressed in 44f8716:

  • [1] sleep defer deadline: the TimeDeltaTrigger sleep now defers with
    timeout=remaining + retry_delay. That always lands after the sleep itself, so it never
    races the final poll, and a lost sleep trigger ends through TaskDeferralTimeout →
    AirflowSensorTimeout in resume_execution, with soft_fail / never_fail applied.
    Pinned in test_execute_complete_redefers_with_capped_sleep_and_started_at and
    test_sleep_capped_to_remaining.
  • [2] timeout is None: agreed, _coerce_timeout makes it unreachable on both the Task
    SDK and 2.11. Removed the branches and test_timeout_attribute_none_never_times_out;
    _remaining_timeout now returns float.
  • [3] naming: _anchor is now _started_at, and the tests use started_at throughout.

Azure sensors suite + msgraph trigger tests: 79 passed. prek and mypy-providers pass.

@dabla

dabla commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Thanks for the review @dabla, all three points addressed in 44f8716:

  • [1] sleep defer deadline: the TimeDeltaTrigger sleep now defers with
    timeout=remaining + retry_delay. That always lands after the sleep itself, so it never
    races the final poll, and a lost sleep trigger ends through TaskDeferralTimeout →
    AirflowSensorTimeout in resume_execution, with soft_fail / never_fail applied.
    Pinned in test_execute_complete_redefers_with_capped_sleep_and_started_at and
    test_sleep_capped_to_remaining.
  • [2] timeout is None: agreed, _coerce_timeout makes it unreachable on both the Task
    SDK and 2.11. Removed the branches and test_timeout_attribute_none_never_times_out;
    _remaining_timeout now returns float.
  • [3] naming: _anchor is now _started_at, and the tests use started_at throughout.

Azure sensors suite + msgraph trigger tests: 79 passed. prek and mypy-providers pass.

Thanks Mathieu, approved, do not forget to resolve conversations if those are addressed, that makes it easier for us to see which comments have been addressed.

@dabla
dabla merged commit 54171ff into apache:main Oct 28, 2026
75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants