Repository navigation
Description
Note
Written by Claude Opus 5.5 (Claude Code) on behalf of Navid
Area
apps/server
Steps to reproduce
On main at e367ef0da1, three kinds of continuation go into ProviderContinuationRequests, a single unbounded queue:
- Claude background-task wakes;
- Codex background-task wakes;
- delegated-completion deliveries, including the re-offers fix(server): wake idle parents for late delegated results #32573 added.
ProviderContinuationService drains that queue with one fiber, one dispatch at a time (requests.take → dispatchContinuation
→ Effect.forever). A wake to a delegated child first takes the parent's dispatch lock (mayStartSubagentFollowup →
withSubagentParentLock).
- A coordinator thread holds its lock for a while. Under load this is common: see A send or queued start to a long thread still reads 6–15 s of its history on main (follow-up to #29468) #36127, and [Bug]: A delegated child's checkpoint capture reads every open delivery of its parent while holding the parent's lock #37393 before fix(server): a delegated child's turn end re-offers only its own parent run's wake #37399 fixed it.
- A continuation for one of its children reaches the head of the queue and waits for that lock.
- Every other continuation on the server, for any thread, waits behind it.
After a restart, recovery re-offers every open completion delivery through the same queue, so wakes also wait behind that wave.
Expected behavior
A wake for one thread is not held up by another thread's lock, or by a backlog of re-deliveries. Order is kept per thread.
Actual behavior
Production server, read-only from its trace and state database:
- One fiber. All 1,189
dispatchContinuationspans in the captured traces belong to one trace, and no two overlap. - Dispatch times. The p50 is 89 ms, but ten dispatches took 9–308 s. Only 2–6 s of each appears in child spans; the rest is a
wait with no span. - A busy fiber. It was busy 300 of 300 seconds in every 5-minute bucket of a 20-minute window.
- No wakes after a restart. For 2 h 40 min, zero background-task wakes were dispatched, while about 1,277 delegated-completion
re-deliveries went through. Before the restart there were 22–54 such wakes per 10 minutes. - Delegated threads working unseen. 140 delegated threads ran on their own, with no T3 run, for a median of 172 minutes.
During that time their MCP writes failed withparent_not_active(delegate_task fails with parent_not_active when a Claude wake turn runs before T3 opens its run #33272).
Impact
On a busy server an agent's wake can take minutes to hours to become a run. Until then the agent's T3 tool calls are refused, and
to its parent it looks idle or dead.
Version or commit
The code path is unchanged on main at e367ef0da1. Measured on a build of main from 11-14 with local performance patches. In
ProviderContinuationService those patches only narrow a read in the retry path; the single consumer and the parent lock are the
same as on main.
Environment
Linux, Node 26, about 185 Claude CLIs, coordinator threads with long-lived delegated threads.
Workaround
None in the app.
Related
- feat(orchestrator): introduce new orchestrator #2829: the queue and its single consumer.
- fix(server): wake idle parents for late delegated results #32573: re-offers of open deliveries.
- fix(server): notify parents after delegated background runs #23542 and fix(server): preserve message order across lineage reads #24442: the parent lock.
- [Bug]: One busy thread's lock stalls every thread's run-end reaction, so queued runs on idle threads do not start #37412: the same single-consumer pattern in the run-end reactor.
- fix(server): MCP tools in a Claude wake turn wait for the turn's run instead of failing with parent_not_active #33330 makes MCP calls in a wake turn wait for the wake's run. That wait is bounded by this dispatcher. delegate_task fails with parent_not_active when a Claude wake turn runs before T3 opens its run #33272 is its report.
We have a fix in our patched build. It does two things:
- It drains the queue per thread family (a delegated child shares its parent's key), on a bounded pool. This is the same shape as
the fix for [Bug]: One busy thread's lock stalls every thread's run-end reaction, so queued runs on idle threads do not start #37412. - It gives provider wakes their own pool, so a restart's re-deliveries cannot hold them up.
In an in-process model of the load above, run on a test clock:
- the longest stretch with no wake dispatched went from 140–170 minutes to none;
- a wake whose family's lock was free was dispatched in about 0.1 s.
I can send the first part as a PR (it builds on the change for #37412), or a maintainer can take it.
Activity