Repository navigation
Resume Strands agents from their last checkpoint when a task retries - #74404
Merged
Merged
Conversation
A task running a Strands Agents agent started the agent over from its prompt on every retry. invoke_durably runs the agent with Strands' checkpointing and a SnapshotSessionManager, saving the conversation and the pause point together at each cycle boundary, so a retry resumes from the last boundary instead. TaskStateStoreStorage keeps that state in the task instance's task state store. AirflowTools now keeps a tool's retry budget across checkpoint resumes, which Strands runs as separate invocations.
kaxil
force-pushed
the
commonai-strands-durability
branch
from
October 8, 2026 13:22
6ab74fa to
974e6c4
Compare
kaxil
marked this pull request as ready for review
October 13, 2026 10:40
kaxil
requested review from
Lee-W,
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl and
potiuk
as code owners
October 13, 2026 10:40
vatsrahul1001
approved these changes
Oct 13, 2026
Lee-W
reviewed
Oct 13, 2026
Lee-W
reviewed
Oct 13, 2026
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A task that runs a Strands Agents agent starts the agent over from its prompt when Airflow retries it: every model call is paid for again and every tool call happens again. The Strands page of the provider docs says so ("There is no step-level replay"). This adds
invoke_durablyinairflow.providers.common.ai.durable.strands, which runs the agent so that a retry resumes it from the last cycle boundary it reached, andTaskStateStoreStorage, a StrandsStoragebacked by the task instance's task state store. #74385 is the Pydantic AI side of the same feature.Design rationale
It uses Strands' own durable-execution contract rather than a per-call journal. Strands pauses an agent built with
checkpointing=Trueat each boundary of a cycle that calls tools (once the model has asked for tools, once they have run), returnsstop_reason="checkpoint", and resumes from a{"checkpointResume": ...}prompt, leaving the conversation to a session manager. Intercepting each model and tool call instead would mean Strands' private middleware (strands._middleware) or subclassingAgentandModel. The cost is granularity: a retry resumes from a cycle boundary, so a task that fails while a cycle's tools run runs that whole tool batch again. The docs say so, and a test pins it.The pause point is saved inside the snapshot, in one write. At each pause,
invoke_durablyputs the checkpoint inagent.stateand callsSnapshotSessionManager.save_snapshot, withsave_latest_on="trigger"so Strands saves nothing on its own beyond redacting or trimming the conversation it holds. Two separate writes could disagree after a crash between them. More importantly, Strands saves the snapshot when an invocation fails, too: amax_tokenscut-off answer, for example, would otherwise become part of the conversation every retry resumes. A test covers that case.invoke_durablytakes a function that builds the agent. Strands restores the snapshot when theAgentis constructed, so the session manager has to go in at construction, and a retry with a different prompt needs a fresh agent after the old session is deleted.functools.partial(Agent, ...)fits theAgentFactoryprotocol.TaskStateStoreStoragekeeps an index of its keys under one extra key. Strands'Storageprotocol needslist(prefix)(delete_sessionuses it), and the task state store accessor only hasget,setanddelete. Keys go under the reserved__commonai_durable__prefix that #74385 also uses, withNEVER_EXPIRE.AirflowToolsno longer resets a tool's retry budget on a resume. Every resume is a new Strands invocation, and the plugin started a fresh budget per invocation, so with checkpointing a tool that fails on every turn would never reach its retry limit. An invocation that brings no new messages (a checkpoint or interrupt resume, or a call with no prompt) now carries on the budget of the one before it.The default session id is the provider's existing
compute_identity_keyover the Dag id, task id, run id and map index, so it is the same on every try and short enough for the 512-character key limit.Gotchas
strands.experimental.checkpoint). It needsstrands-agents1.51 or later, which the docs' existing>=1.56.0install line covers. Strands is not inuv.lock, so the new tests run in the agent-framework CI job, which this adds them to.invoke_durablyreturns at the agent's first stop that is not a pause (its answer, an interrupt, a cancellation) and deletes the saved state then, not when the task succeeds. A failure later in the task, such as in a second agent, runs the finished agent again on the retry. Add AirflowDurability capability for durable agent replay across task retries #74385 behaves the same way in a plain@task.SandboxToolsetwithoutattach_tois destroyed when the try ends, so a resumed agent's conversation describes files the retry's new sandbox does not have.AirflowToolskeeps only the converted tools, soinvoke_durablycannot detect this the wayAgentOperator(durable=True)refuses a sandbox; the docs say it instead.model.stateful) raisesValueError, since Strands drops a restored conversation for it.[workers] state_store_backendis set, and grows with it.S3Storageasstorage=.Follow-ups
TaskStateStoreStoragedrop its key index.End-to-end run
A test Dag ran a Strands agent with a scripted model in a
@task(retries=1). Cycle 1 called a tool that counts its calls in a file; cycle 2's model call failed on try 1 only. Try 2 logged the resume and finished, the tool ran once in total, and the task state store held no rows for the task afterwards.