You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(core): evented workflow engine for background tasks + suspend/resume (#16260)
## Summary
- Background tasks now execute as evented workflows (each task is a
registered `__background-task` workflow run). The legacy in-manager
dispatch loop is preserved behind `engine: 'legacy'`.
- Adds suspend/resume to background tasks. Tools can call
`ctx.agent.suspend(data)` from inside `execute` to park their bg task;
resume with `mastra.backgroundTaskManager.resume(taskId, resumeData)` or
`agent.resumeStreamUntilIdle(resumeData, { runId, toolCallId })`.
- New `agent.resumeStreamUntilIdle()` (and corresponding HTTP routes /
`@mastra/client-js` methods) resumes a suspended agent stream and keeps
the SSE stream open through bg-task completion or suspension and the
agent's follow-up turn.
- Storage adapters track two new fields: `suspendPayload` (JSON) and
`suspendedAt` (timestamp). SQL adapters auto-migrate via
`alterTable.ifNotExists`.
## Why
The previous in-manager dispatch loop owned executor invocation,
retries, and lifecycle directly. That made suspend/resume difficult —
there was no durable cursor to park the run on. Routing each task
through an evented workflow gives us snapshotting + resume for free, and
lets `agent.streamUntilIdle()` close cleanly when a bg task suspends
rather than waiting indefinitely for completed/failed/cancelled.
Pubsub topics, lifecycle event shapes, concurrency gating, and the
`BackgroundTaskManager.stream()` contract are unchanged — the workflow
engine is plumbed underneath the existing seams.
## Migration notes
- **Default engine flip.** Background-tasks default to `engine:
'workflow'`. Set `backgroundTasks: { engine: 'legacy' }` to keep the
previous in-manager dispatch loop. `manager.resume(...)` only works with
the workflow engine.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5
Background tasks can now pause themselves, save their state, and be
resumed later (manually or via a streaming agent). Tasks run as durable
evented workflows for reliability and resumeability while the old
dispatch loop remains available behind a config flag.
---
## Overview
Runs background tasks as evented workflow runs (registered as the
__background-task workflow) and adds durable suspend/resume semantics.
The workflow engine is integrated under existing seams so pubsub topics,
lifecycle event shapes, concurrency gating, and
BackgroundTaskManager.stream() remain compatible. The legacy in-manager
dispatch loop is preserved behind backgroundTasks.engine = 'legacy'.
## Key Changes
- Workflow-backed background tasks
- New BACKGROUND_TASK_WORKFLOW_ID ('__background-task') and
buildBackgroundTaskWorkflow(manager) implementing retries, timeouts,
aborts, progress throttling, persistence, and durable suspend/resume.
- Internal workflow resolution updated so Mastra can register/use
internal workflows.
- Suspend / resume
- Tools may call ctx.agent.suspend(data) inside execute to persist
suspendPayload, set suspendedAt, snapshot execution, and release
concurrency slots.
- Resume via mastra.backgroundTaskManager.resume(taskId, resumeData)
(workflow engine only) or agent.resumeStreamUntilIdle(resumeData, {
runId, toolCallId }) for in-band streaming resumes.
- BackgroundTaskHandle gains checkIfSuspended(...) and resume(...).
- Streaming API
- New Agent API/method: resumeStreamUntilIdle(...) plus server route
POST /api/agents/:agentId/resume-stream-until-idle and client-js helper.
Keeps SSE open through resume completion/suspension and the agent’s
follow-up turn.
- streamUntilIdle/resume flows treat background-task-suspended as
terminal-for-wrapper (enqueued so callers can trigger continuations).
- Lifecycle & events
- New stream chunk types: background-task-suspended and
background-task-resumed with payload shapes.
- BackgroundTaskManager: new async resume(taskId, resumeData?),
runLocalSuspendHooks, runLocalCompletionHooks; publishLifecycleEvent now
accepts suspended/resumed types. taskContexts and activeAbortControllers
exposed for workflow steps.
- Continuation dedup strengthened: processed terminal events tracked by
`${taskId}:${chunkType}` to avoid suspend→resume→complete collisions.
- stream/continuation logic updated to track suspended vs running
tool-call IDs.
- Storage, schema & adapters
- Added suspend_payload (suspendPayload) and suspendedAt (timestamp) to
background-tasks schema; many adapters auto-migrate via
alterTable.ifNotExists.
- Listing/deletion now support toolCallId filtering and
suspendedAt-aware date filtering/ordering across adapters (Postgres,
MSSQL, LibSQL, ClickHouse, Cloudflare D1/DO/KV, DynamoDB, Convex, Lance,
MongoDB, Upstash, etc.).
- In-memory and other stores updated to parse/serialize suspendPayload
and suspendedAt.
- Types, client libs & UI
- BackgroundTaskStatus includes 'suspended'; BackgroundTask gains
suspendPayload?: unknown and suspendedAt?: Date.
- MastraClient.listBackgroundTasks accepts optional toolName and
toolCallId filters.
- client-js: route/typing for resume-stream-until-idle and
resumeStreamUntilIdle client method (maxIdleMs option).
- React/toUIMessage and MastraUIMessageMetadata updated to surface
suspendedAt and restore suspend metadata after resume.
- Playground UI components updated to display suspend payload and
suspendedAt; some component prop shapes simplified to pass a
backgroundTask object.
- Examples & tests
- Example tool cryptoResearchTool converted to a suspend/resume workflow
(approval flow); some agents set autoResumeSuspendedTools.
- Extensive unit/e2e tests added/updated to start/stop the event engine
and exercise suspend/resume, resume-stream-until-idle, and deterministic
manager behavior.
- New workflow registration tests verify internal workflow registration
behavior.
## Notable public API / signature changes
- New exports/types: BACKGROUND_TASK_WORKFLOW_ID;
BackgroundTaskSuspended/Resumed chunk payloads; CheckIfSuspendedPayload.
- BackgroundTaskManager: new async resume(taskId, resumeData?) and
public runLocalSuspendHooks/runLocalCompletionHooks; taskContexts and
activeAbortControllers are no longer private.
- Agent: added resumeStreamUntilIdle overloads (streaming resume path).
- BackgroundTaskHandle: checkIfSuspended(...) and resume(...) methods.
- client-js: new resumeStreamUntilIdle method and route types;
MastraClient.listBackgroundTasks query supports toolName/toolCallId.
- React/playground components: BackgroundTaskMetadataDialogTrigger prop
shape changed to accept a backgroundTask object.
## Docs
- docs: added “Suspending and resuming” section, examples, and mention
of workflow engine requirement.
- ChunkType docs updated with background-task-suspended/resumed
payloads.
- client-js docs updated with resumeStreamUntilIdle usage.
## Migration & compatibility
- Default backgroundTasks.engine is now 'workflow'. Set backgroundTasks:
{ engine: 'legacy' } to retain previous dispatch loop.
- manager.resume(...) only works with the workflow engine.
- Pubsub topics, lifecycle chunk shapes, concurrency gating, and
BackgroundTaskManager.stream() contract remain backward-compatible.
- Storage migrations run automatically on init for adapters that support
alterTable.ifNotExists().
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Background tasks now run as evented workflows. Each task is dispatched as a workflow run that owns executor invocation, retries, and suspend/resume; pubsub topics, lifecycle event shapes, concurrency gating, and the `BackgroundTaskManager.stream()` contract are unchanged.
6
+
7
+
Add suspend/resume to background tasks. Tools can call `ctx.agent.suspend(data)` from `execute` to pause a task and release the concurrency slot; resume with `mastra.backgroundTaskManager.resume(taskId, resumeData)` or `agent.resumeStreamUntilIdle(resumeData, { runId, toolCallId })`. Surfaces `background-task-suspended` / `background-task-resumed` chunks on `backgroundTaskManager.stream()` and `agent.streamUntilIdle().fullStream`.
0 commit comments