[deepujain]

Title: Stop retries when the retry callback raises AirflowFailException

Summary

A retry callback can decide that another attempt would be too late or too costly. Raising AirflowFailException there currently logs a callback error and retries the task anyway. This change lets the callback stop retries and run normal failure handling with the original task exception.

Changes

Retry callbacks run before the task outcome is reported. If one raises AirflowFailException, the remaining retry callbacks stop; the failure callback, listener and failure email see the original task failure. Ordinary callback errors still allow retries.

The supervisor starts its existing finalization timeout before the retry callback runs and retains the original retry payload until an outcome arrives. If the callback process exits early, that payload remains available; an explicit outcome or server termination takes precedence. The schema snapshots and generated Go/TypeScript consumers include the new message.

Related: #60172. Earlier attempts #60415, #64198, #66781 and #73849 closed without merging; #63342 was an empty closed PR. This patch addresses the retry-callback case. Losing a final outcome message remains a separate recovery limitation.

Test plan

• The same regression fails on unchanged main because the task remains UP_FOR_RETRY, and passes with this patch. An ordinary callback error remains a retry.
• Full Task SDK suite (Python 3.12): 4,399 passed, 6 skipped. Both changed core protocol inventory tests passed. The current-runtime provider-registration smoke suite passed all 30 tests after the base refresh.
• Configured local-bundle dag.test() replay: the stopping callback fails after attempt 1; an ordinary callback error allows attempt 2; a task raising AirflowFailException directly fails after attempt 1.
• Callback lifecycle tests cover early process exits, explicit failure, server termination, IPC send errors and shared finalization overtime.
• breeze testing core-tests with the selected non-DB groups: 5,714 passed, 11 skipped, 2 expected failures. All six selected SQLite DB groups also passed: 12,846 passed, 92 skipped.
• Regular and manual prek checks passed, including type checks and generated-schema checks. Explicit prek run mypy-providers --stage manual --all-files also passed.
• go test ./... and go test -race ./... passed. TypeScript SDK: 961 tests passed.
• TypeScript documentation passed using a clean source copy with its own Linux dependencies.
• Selected provider DB groups passed. The broad provider non-DB run had 28,674 passes, 7 failures and 37 errors during worker/resource failures, collection failures, a Glue assertion and Mongo fixture setup failures. Serial recovery of every failed case and uncollected module passed: 87 tests. Mongo recovery used Docker Desktop's supported TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal; Ryuk remained enabled. The original broad invocation was not green.
• Python documentation: all 116 package builds completed successfully across the full build and scoped recovery runs. The original broad build and its serial continuation were OOM-killed; seven fresh-process batches then completed the 49 unfinished packages. Those failed invocations remain failed.
• Hosted CI and human review remain pending.

The full SDK suite, configured Dag replay and benchmark ran before the latest provider-only base refresh. The Task SDK sources, issue diff and resolved dependency versions are unchanged by that refresh; the provider-registration smoke suite above ran afterward.

Finalization cost

A local MsgPack socket benchmark alternated five base/candidate runs of 5,000 iterations with a no-op callback. The extra supervision message adds about 11.5 microseconds to retry finalization with a callback. It lets the callback decide the outcome while preserving bounded finalization. The unchanged parsing control moved 3.5%, below the documented 10% noise threshold.

Path	Base median	Candidate median
Retry finalization with callback	20.95 µs	32.43 µs
Success finalization	12.65 µs	12.98 µs
JSON parsing control	9.43 µs	9.76 µs

closes: #60172

---

Was generative AI tooling used to co-author this PR?

• Yes (OpenAI Codex)

Generated-by: OpenAI Codex following the guidelines