Repository navigation
Fix MSGraphSensor timeout being reset on every re-defer - #75774
Conversation
A defer timeout becomes an absolute deadline of now + timeout, and MSGraphSensor re-deferred with the full sensor timeout between polls (while the request defer carried none), so the sensor never timed out. Carry the first execute's start time in next_kwargs, give each request defer the remaining time, and raise AirflowSensorTimeout after a not-done poll once the timeout has elapsed, so soft_fail and never_fail apply as for core sensors. Stop passing the sensor timeout to MSGraphTrigger, where it was used as the HTTP request timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dabla
left a comment
There was a problem hiding this comment.
Related: #62241 (closed #62157 with the sliding re-defer timeout this PR replaces); PowerBI's PowerBITrigger enforces its timeout inside the trigger, so it was never affected.
Addressed
Nothing since the previous round: no reviews or threads on the PR yet.
Open
- The premise of the PR holds, and it is the issue you remember. #62241 (March 2026, "Pass timeout to defer() in MSGraphSensor", closing #62157) put
timeout=self.timeouton the sleep re-defer only. Core stores a deferral deadline asutcnow() + timeouton every defer and clears it on a defer without one (execution APIti_update_state,TaskInstance._defer_taskon 2.x), so that fix restarted the clock on every poll and theMSGraphTriggerdefer in between wiped it: the sensor still never timed out. PowerBI is a different mechanism:PowerBITriggerpolls inside the trigger with atime.monotonic()loop bounded by itstimeout, so its deadline never depended on re-defers and was not touched by #62241 or by #74675. This PR is the first fix of the MSGraph sensor itself. - [1] warning: the sleep defer now has no deadline at all; keep one above the sleep length.
- [2] nit:
timeout is Nonebranches and their test are unreachable through the base class. - [3] nit:
_anchor/started_at/sensor_started_atname one concept. - Not commented:
tests_commondeferrable_operatordropstask.kwargswhen it resumes, so the three unchanged end-to-end tests exercise the "missing start time" fallback, not the carriedsensor_started_at; the kwargs round trip is covered bytest_timeout_via_resume_executionand the direct unit tests, so no gap in coverage, only a harness limitation worth knowing. Pre-existing and out of scope:execute_completereturnsNone(success) when the event carries an emptyresponse.
Head 3574742 reviewed in diff mode. Verified against the Task SDK and 2.11: BaseSensorOperator.resume_execution catches AirflowException (so AirflowSensorTimeout raised from the resumed method honours soft_fail on both, never_fail on 3.1+), self.timeout is always a number, AirflowSensorTimeout and timezone are in common.compat.sdk since before the 1.23.0 floor, datetime.UTC is fine with the provider's requires-python >= 3.11, time-machine is a devel-common dependency. Audits run: parity with BaseSensorOperator.execute (poke, timeout check, sleep, same message), lifecycle and timeout (per-defer deadlines on both deferral paths, which produced [1]), repeated side effects (retry_execute no longer re-enters execute), SDK subclass contract (produced [2]), docs and test wording, python checklist (vocabulary, [3]). No tests run locally; the PR body reports 80 passed on the azure sensors and msgraph trigger suites.
Give the TimeDeltaTrigger sleep a defer timeout of the remaining time plus retry_delay, so a lost sleep trigger still ends in a sensor timeout. Drop the unreachable timeout=None branches and use started_at consistently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review @dabla, all three points addressed in 44f8716:
Azure sensors suite + msgraph trigger tests: 79 passed. prek and mypy-providers pass. |
Thanks Mathieu, approved, do not forget to resolve conversations if those are addressed, that makes it easier for us to see which comments have been addressed. |
Fix MSGraphSensor timeout being reset on every re-defer
What
MSGraphSensorcould never time out. It now raisesAirflowSensorTimeoutoncetimeoutseconds have passed since the first
execute()of the try, sosoft_fail/never_failwork again. It also stops passing the sensor timeout to
MSGraphTrigger, which used it asthe HTTP request timeout.
Why
A defer timeout is stored as
utcnow() + timeout(AF3 execution APIti_update_state,AF2
TaskInstance._defer_task), andtimeout=Noneclears any earlier deadline. The sensoralternated between an
MSGraphTriggerdefer without a timeout (clearing the deadline) and aTimeDeltaTriggersleep with the full timeout (restarting it), so the deadline moved forwardon every poll and was never reached.
Separately,
timeout=self.timeoutwas passed toMSGraphTrigger, where it became the httpxper-request timeout (7 days by default).
How
execute()records its start time and carries it innext_kwargs(
sensor_started_at) throughexecute_completeandretry_execute; if it is missing(TIs deferred before upgrading), it falls back to now.
bounded as well.
BaseSensorOperator.execute. Past the deadline,AirflowSensorTimeoutis raised from theresumed method, so core
resume_executionstill appliessoft_fail(andnever_failon3.1+). It is not retried.
poll, which is given at least
retry_delayso that small clock differences between workerscannot cut it short. The sensor can therefore overrun its deadline by at most
retry_delay.A done response that arrives after the deadline still succeeds.
retry_delay, so a lost sleeptrigger still ends in
AirflowSensorTimeout.retry_executenow defers directly instead of callingexecute().The HTTP timeout stays at the trigger/hook default (
None); each request is still bounded bythe remaining sensor time. A dedicated HTTP-timeout parameter can follow if needed.
How tested
None), deadline not moving across re-defers, sleep capped at the remaining time,sleep defer deadline,
AirflowSensorTimeoutpast the deadline, one last poll (given at leastretry_delay) whenresuming near, at or past the deadline, done response after the deadline,
resume_executionwith no flag /soft_fail/never_fail, missing-start-time fallback.reverted, the new sensor tests fail.
prek run --files <changed files>and mypy-providers pass.🤖 Generated with Claude Code